Communication system and method thereof

ABSTRACT

The present invention makes it possible to process at high speed the foreground component image and the background component image of picked up images on a network platform. A client computer  27  outputs information specifying image data desired to separate to a separation server  11.  The separation server  11  obtains the specified image data from a storage server  18  and outputs it to a motion detecting server  12  to perform motion detection processing. Thereafter, the image data, motion vector and positional information are output to an area specifying server  13.  The area specifying server  13  generates area information of the image data and outputs the area information to a mixture ratio calculating server  14  in addition to the image data, the motion vector and the positional information. The mixture ratio calculating server  14  calculates a mixture ratio on the basis of the image data, the motion vector, the positional information and the area information, and a foreground/background image separation server  15  separates foreground and background of the input image on the basis of such information. The present invention may be employed in a business model for image processing.

TECHNICAL FIELD

The present invention relates to a communication system and a methodthereof, and more particularly, to a communication system and a methodfor distributing image processing to a plurality of servers on a networkto perform separated or isolated image processing desired by a user,wherein the communication system and the method make it possible toimprove the speed of image processing on a network platform at reducedcosts.

BACKGROUND ART

In general, techniques for synthesizing images desired by a user on anetwork are widely known.

Synthesized image is usually generated by overlapping and joining anumber of pre-existing images such as images having already been storedin a server connected through a network or images having been madeavailable from digital still cameras. In some cases, the synthesizedimage may be generated by a texture mapping.

However, there are problems in that synthesized images generated byusing the above conventional methods give rise to the occurrences ofmotion blur. For example, motion blur commonly occurs with a moving bodythat cannot be accurately adjusted into the synthesized images.Consequently, synthesizing processes are performed with low accuracy andunnatural appearances are frequently generated in the synthesizedimages.

Additionally, if one server performs synthesizing process in a multiplebundle task, a significant duration of time is required for the process.For example, even when the same background images are to be overlapped,the overlapping process is repeated as many times as the number ofimages desired. This poses a problem to users in that when theprocessing services are charged by time, the cost can be a considerableburden.

DISCLOSURE OF INVENTION

In view of the foregoing, it is an object of the present invention toprovide a communication system and a method for distributing imageprocessing characterized by having multiple tasks to a plurality ofservers on a network to perform separated or isolated image processingdesired by a user, wherein the communication system and the method makeit possible to improve the speed of image processing on a networkplatform at reduced costs.

A first communication apparatus of the present invention comprises aseparating means for separating a predetermined image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined every pixel in accordance with an amount oflight forming an image integrated over time, into a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object; a requested informationinput means for inputting requested information of a user; and anencoded data output means for outputting encoded data generated when theimage is separated into a foreground component image and a backgroundcomponent image by the separating means on the basis of the requestedinformation input by the requested information input means.

A charging means may be further comprised for performing a chargeprocessing in accordance with the requested information.

The charging means may generate charge information including a user ID,a communication system ID and cost information corresponding to therequested information in accordance with the requested information.

The charging means may perform a charge processing with respect to afinancial account of the user on the basis of the charge information.

The charging means may perform the charge processing by deducting thenumber of points corresponding to the cost information from the numberof points of each user used in the charge processing.

The encoded data output means may output the encoded data in a wayobtainable only to a user having finished the charge processing afterthe charge processing has been finished by the charging means.

The requested information input means may input a predetermined imagedata in addition to the requested information of a user. The encodeddata output means may output the encoded data as a significantinformation generated when the image is separated into the foregroundcomponent image and the background component image by the separatingmeans on the basis of the requested information input by the requestedinformation input means and the predetermined image data, in a wayobtainable only to the user having finished the charge processing afterthe charge processing has been finished by the charging means.

An area information generating means for generating area information fordiscerning any of a foreground area having foreground object componentsconstituting the foreground object of the predetermined image data, abackground area having background object components constituting thebackground object of the predetermined image data and a mixed area inwhich the foreground area and the background area are mixed may befurther comprised. The significant information may include the areainformation and the encoded data output means may output the encodeddata as the area information generated when the image is separated intothe foreground component image and the background component image by theseparating means on the basis of the requested information input by therequested information input means and the predetermined image data, in away obtainable only to the user after the charge processing has beenfinished by the charging means.

A mixture ratio generating means for generating a mixture ratioindicating a ratio in which the foreground area and the background areaare mixed in the mixed area of the predetermined image data may befurther comprised. The significant information may include the mixtureratio and the encoded data output means may output the encoded data asthe mixture ratio generated when the image is separated into theforeground component image and the background component image by theseparating means on the basis of the requested information input by therequested information input means and the predetermined image data, in away obtainable only to the user after the charge processing has beenfinished by the charging means.

The separating means may separate a predetermined image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined every pixel in accordance with an amount oflight forming a predetermined image integrated over time, into aforeground component image having foreground object componentsconstituting the foreground object and a background component imagehaving background object components constituting the background objecton the basis of the area information and the mixture ratio. Thesignificant information may include the foreground component image andthe background component image and the encoded data output means mayoutput the requested information input by the requested informationinput means and the encoded data as the foreground component image andthe background component image generated when the image is separatedinto the foreground component image and the background component imageby the separating means, in a way obtainable only to the user after thecharge processing has been finished by the charging means.

The requested information input means may input an image data ID fordiscerning a predetermined image data in addition to the requestedinformation of a user, and the encoded data output means may output theencoded data as the significant information generated when thepredetermined image is separated into the foreground component image andthe background component image by the separating means on the basis ofthe requested information input by the requested information input meansand the predetermined image data corresponding to the image data ID, ina way obtainable only to the user after the charge processing has beenfinished by the charging means.

An area information generating means for generating area information fordiscerning any of a foreground area having foreground object componentsconstituting the foreground object of the predetermined image datacorresponding to the image data ID, a background area having backgroundobject components constituting the background object of thepredetermined image data corresponding to the image data ID and a mixedarea in which the foreground area and the background area are mixed maybe further comprised. The significant information may include the areainformation and the encoded data output means may output the encodeddata as the area information generated when the predetermined image isseparated into the foreground component image and the backgroundcomponent image by the separating means on the basis of the requestedinformation input by the requested information input means and thepredetermined image data, in a way obtainable only to the user after thecharge processing has been finished by the charging means.

A mixture ratio generating means for generating a mixture ratio of themixed area in the predetermined image data corresponding to the imagedata ID may be further comprised. The significant information mayinclude the mixture ratio and the encoded data output means may outputthe encoded data as the mixture ratio generated when the predeterminedimage is separated into the foreground component image and thebackground component image by the separating means on the basis of therequested information input by the requested information input means andthe predetermined image data corresponding to the image data ID, in away obtainable only to the user after the charge processing has beenfinished by said charging means.

The separating means may separate a predetermined image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined every pixel in accordance with an amount oflight forming an image integrated over time, into a foreground componentimage having foreground object components constituting the foregroundobject and a background component image having background objectcomponents constituting the background object on the basis of the areainformation and the mixture ratio. The significant information mayinclude the foreground component image and the background componentimage and the encoded data output means may output the requestedinformation input by the requested information input means and theencoded data as the foreground component image and the backgroundcomponent image generated when a predetermined image is separated intothe foreground component image and the background component image by theseparating means, in a way obtainable only to a user after the chargeprocessing has been finished by the charging means.

A first communication method of the present invention comprises: aseparating step of separating a predetermined image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined every pixel in accordance with an amount oflight forming an image integrated over time, into a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object; a requested informationinput step of inputting requested information of a user; and an encodeddata output step of outputting encoded data generated when an image isseparated into the foreground component image and the backgroundcomponent image in the separating step on the basis of the requestedinformation input in the requested information input step.

A program of a first recording medium of the present inventioncomprises: a separation control step of controlling separation of animage data that is obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values determined every pixel inaccordance with an amount of light forming an image integrated overtime, into a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject; a requested information input control step of controlling inputof requested information of a user; and an encoded data output controlstep of controlling output of encoded data generated when an image isseparated into the foreground component image and the backgroundcomponent image in the separation control step on the basis of therequested information input in the requested information input controlstep.

A first program of the present invention makes a computer perform: aseparation control step of controlling separation of a predeterminedimage data that is obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values determined every pixel inaccordance with an amount of light forming an image integrated overtime, into a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject; a requested information input control step of controlling inputof requested information of a user; and an encoded data output controlstep of controlling output of encoded data generated when an image isseparated into the foreground component image and the backgroundcomponent image in the separation control step on the basis of therequested information input in said requested information input controlstep.

In the first communication system, the method thereof and the firstprogram, an image data that is obtained by an image pickup elementhaving a predetermined number of pixels, each having a time integrationeffect, and that is comprised of pixel values determined every pixel inaccordance with an amount of light forming an image integrated over timeis separated into a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject; requested information of a user is input; and encoded datagenerated when an image is separated into the foreground component imageand the background component image on the basis of the requestedinformation input is output.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a configuration view of an embodiment of an image processingsystem to which the present invention is applied;

FIG. 2 is a configuration view of a separation server in FIG. 1;

FIG. 3 is a configuration view of a camera terminal unit in FIG. 1;

FIG. 4 is a functional view of the separation server in FIG. 1;

FIG. 5 is another functional view of the separation server in FIG. 1;

FIG. 6 is a functional view of a motion detecting server in FIG. 1;

FIG. 7 is another functional view of the motion detecting server in FIG.1;

FIG. 8 is a functional view of an area specifying server in FIG. 1;

FIG. 9 is another functional view of the area specifying server in FIG.1;

FIG. 10 is a functional view of a mixture ratio calculating server inFIG. 1;

FIG. 11 is another functional view of the mixture ratio calculatingserver in FIG. 1;

FIG. 12 is a functional view of a foreground/background image separatingserver in FIG. 1;

FIG. 13 is another functional view of the foreground/background imageseparating server in FIG. 1;

FIG. 14 is a functional view of a motion blur adjusting server in FIG.1;

FIG. 15 is another functional view of the motion blur adjusting serverin FIG. 1;

FIG. 16 is a functional view of an encoding server in FIG. 1;

FIG. 17 is another functional view of the encoding server in FIG. 1;

FIG. 18 is a functional view of a storage server in FIG. 1;

FIG. 19 is another functional view of the storage server in FIG. 1;

FIG. 20 is a functional view of a synthesizing server in FIG. 1;

FIG. 21 is another functional view of the synthesizing server in FIG. 1;

FIG. 22 is a functional view of a correcting server in FIG. 1;

FIG. 23 is another functional view of the correcting server in FIG. 1;

FIG. 24 is a functional view of a purchasing server in FIG. 1;

FIG. 25 is a functional view of a selling server in FIG. 1;

FIG. 26 is a functional view of a retrieving server in FIG. 1;

FIG. 27 is a block diagram showing the separation server;

FIG. 28 is a diagram illustrating image pickup by a sensor;

FIG. 29 is a diagram illustrating an arrangement of pixels;

FIG. 30 is a diagram illustrating operation of a detecting element;

FIG. 31A is a diagram illustrating an image obtained by picking up animage of an object corresponding to a moving foreground and an objectcorresponding to a stationary background;

FIG. 31B is another diagram illustrating an image obtained by picking upan image of an object corresponding to a moving foreground and an objectcorresponding to a stationary background;

FIG. 32 is a diagram illustrating background area, foreground area,mixed area, covered background area and uncovered background area;

FIG. 33 is a modeling diagram in which pixel values of pixels adjacentto each other in a row are expanded in time direction in picking up animage of an object corresponding to a stationary foreground and anobject corresponding to a stationary background;

FIG. 34 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 35 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 36 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 37 is a diagram illustrating an example in which pixels offoreground area, background area and mixed area are extracted;

FIG. 38 is a diagram illustrating relationship of pixels and a model inwhich pixel values are expanded in the time direction;

FIG. 39 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 40 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 41 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 42 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 43 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 44 is a flowchart illustrating a process of adjusting the amount ofmotion blur.

FIG. 45 is a block diagram illustrating a configuration example of anarea specifying unit 103;

FIG. 46 is a diagram illustrating an image when an object correspondingto foreground is being moved;

FIG. 47 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 48 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 49 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 50 is a diagram illustrating conditions of an area determination;

FIG. 51A is a diagram illustrating an example of an area specificationresult in the area specifying unit 103;

FIG. 51B is a diagram illustrating an example of an area specificationresult in the area specifying unit 103;

FIG. 51C is a diagram illustrating an example of an area specificationresult in the area specifying unit 103;

FIG. 51D is a diagram illustrating an example of an area specificationresult in the area specifying unit 103;

FIG. 52 is a diagram illustrating an example of an area specificationresult in the area specifying unit 103;

FIG. 53 is a flowchart illustrating an area specification processing;

FIG. 54 is a block diagram of another example of a configuration of thearea specifying unit 103;

FIG. 55 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 56 is a diagram illustrating an example of a background image;

FIG. 57 is a block diagram illustrating a configuration of a binaryobject image extracting portion 302;

FIG. 58A is a diagram illustrating calculation of correlation value;

FIG. 58B is a diagram illustrating calculation of correlation value;

FIG. 59A is a diagram illustrating calculation of correlation value;

FIG. 59B is a diagram illustrating calculation of correlation value;

FIG. 60 is a diagram illustrating an example of a binary object image;

FIG. 61 is a block diagram illustrating a configuration of a time changedetector 303;

FIG. 62 is a diagram illustrating a determination in an area determiningportion 342;

FIG. 63 is a diagram illustrating an example of a determination in thetime change detector 303;

FIG. 64 is a flowchart illustrating an area specification processing inthe area determining unit 103;

FIG. 65 is a flowchart illustrating details of the area specificationprocessing;

FIG. 66 is a block diagram of another configuration of the areaspecifying unit 103;

FIG. 67 is a block diagram illustrating a configuration of a robustprocessing portion 361;

FIG. 68 is a diagram illustrating motion compensation in a motioncompensator 381;

FIG. 69 is a diagram illustrating motion compensation in the motioncompensator 381;

FIG. 70 is a flowchart illustrating the area specification processing;

FIG. 71 is a flowchart illustrating details of a robust processing;

FIG. 72 is a block diagram illustrating an example of a configuration ofa mixture ratio calculating unit 104;

FIG. 73 is a diagram illustrating an example of an ideal mixture ratioα;

FIG. 74 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 75 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 76 is a diagram illustrating an approximation using correlation offoreground components;

FIG. 77 is a diagram illustrating the relationship of C, N and P;

FIG. 78 is a block diagram illustrating a configuration of an estimatedmixture ratio processor 401;

FIG. 79 is a diagram illustrating an example of an estimated mixtureratio;

FIG. 80 is a block diagram illustrating another configuration of themixture ratio calculator 104;

FIG. 81 is a flowchart illustrating a mixture ratio calculationprocessing;

FIG. 82 is a flowchart illustrating an estimated mixture ratiocalculation processing;

FIG. 83 is a diagram illustrating a linear line approximation of themixture ratio α;

FIG. 84 is a diagram illustrating a plane approximation of the mixtureratio α;

FIG. 85 is a diagram illustrating correspondence of pixels in aplurality frames when the mixture ratio α is calculated;

FIG. 86 is a block diagram illustrating another configuration of themixture ratio estimating processor 401;

FIG. 87 is a diagram illustrating an example of an estimated mixtureratio;

FIG. 88 is a flowchart illustrating the mixture ratio estimationprocessing by using a model corresponding to a covered background area;

FIG. 89 is a block diagram illustrating an example of a configuration ofa foreground/background separator 105;

FIG. 90A is a diagram showing an input image, a foreground componentimage and a background component image;

FIG. 90B is a diagram showing an input image, a foreground componentimage and a background component image;

FIG. 91 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 92 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 93 is a modeling diagram in which pixel values are expanded in thetime direction and shutter time which is divided into a time interval;

FIG. 94 is a block diagram illustrating an example of a configuration ofa separating portion 601;

FIG. 95A is a diagram showing an example of a foreground component imageand a background component image separated;

FIG. 95B is a diagram showing an example of a foreground component imageand a background component image separated;

FIG. 96 is a flowchart illustrating a foreground/background separationprocessing;

FIG. 97 is a block diagram showing an example of a configuration of amotion blur adjusting unit 106;

FIG. 98 is a diagram illustrating a unit of processing;

FIG. 99 is a modeling diagram in which pixel values of a foregroundcomponent image are expanded in the time direction and shutter timewhich is divided into a time interval;

FIG. 100 is a modeling diagram in which pixel values of foregroundcomponent image are expanded in the time direction and shutter timewhich is divided into a time interval;

FIG. 101 is a modeling diagram in which pixel values of a foregroundcomponent image are expanded in the time direction and shutter timewhich is divided into a time interval;

FIG. 102 is a modeling diagram in which pixel values of a foregroundcomponent image are expanded in the time direction and shutter timewhich is divided into a time interval;

FIG. 103 is a diagram showing another configuration of the motion bluradjusting unit 106;

FIG. 104 is a flowchart illustrating a process of adjusting the amountof motion blur contained in a foreground component image in the motionblur adjusting unit (106);

FIG. 105 is a block diagram showing another example of the configurationof the motion blur adjusting unit 106;

FIG. 106 is a diagram showing an example of a model in which therelationship of pixel values and foreground components is specified;

FIG. 107 is a diagram illustrating a calculation of a foregroundcomponent;

FIG. 108 is a diagram illustrating a calculation of a foregroundcomponent;

FIG. 109 is a flowchart illustrating the motion blur eliminationprocessing of a foreground;

FIG. 110 is a block diagram showing another configuration of a functionof the separation server; FIG. 111 is a diagram showing a configurationof a synthesizer 1001;

FIG. 112 is a block diagram showing another configuration of a functionof the separation server;

FIG. 113 is a block diagram showing a configuration of a mixture ratiocalculator 1101;

FIG. 114 is a block diagram showing a configuration of aforeground/background separator 1102;

FIG. 115 is a block diagram showing another configuration of a functionof the separation server;

FIG. 116 is a diagram showing a configuration of a synthesizer 1201;

FIG. 117 is a flowchart illustrating a separating service;

FIG. 118 is a flowchart illustrating a charge processing;

FIG. 119 is a diagram illustrating a charge processing;

FIG. 120 is a flowchart illustrating another example of the chargeprocessing;

FIG. 121 is a flowchart illustrating a motion detecting service;

FIG. 122 is a flowchart illustrating an area specifying service;

FIG. 123 is a flowchart illustrating a mixture ratio calculatingservice;

FIG. 124 is a flowchart illustrating a foreground/background separatingservice;

FIG. 125 is a flowchart illustrating a motion blur adjusting service;

FIG. 126 is a diagram illustrating an encoding server;

FIG. 127 is a flowchart illustrating an encoding service;

FIG. 128 is a diagram illustrating a compressing ability by a means ofthe encoding processing;

FIG. 129 is a diagram illustrating another example of the encodingserver;

FIG. 130 is a flowchart illustrating a synthesizing service;

FIG. 131 is a diagram illustrating an encrypting motion blur addingunit;

FIG. 132 is a diagram illustrating an encrypting motion blur eliminatingunit;

FIG. 133 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 134 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 135 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 136 is a flowchart illustrating an encryption processing;

FIG. 137 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 138 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 139 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 140 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 141 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 142 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 143 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 144 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 145 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 146 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 147 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 148 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 149 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 150 is a diagram illustrating a process of adding an encryptingmotion blur;

FIG. 151 is a flowchart illustrating an encrypting service;

FIG. 152 is a diagram illustrating a correcting server;

FIG. 153A is a diagram illustrating an adjusting process;

FIG. 153B is a diagram illustrating an adjusting process;

FIG. 153C is a diagram illustrating an adjusting process;

FIG. 153D is a diagram illustrating an adjusting process;

FIG. 154 is a flowchart illustrating an adjusting process;

FIG. 155 is a flowchart illustrating a purchasing service;

FIG. 156 is a flowchart illustrating a selling service;

FIG. 157 is a flowchart illustrating a charging service;

FIG. 158 is a diagram illustrating the retrieving server;

FIG. 159 is a flowchart illustrating a retrieving service; and

FIG. 160 is a diagram illustrating a retrieving service.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 1 is a drawing showing an embodiment of an image processing systemaccording to the present invention.

The image processing system of the present invention is constructed suchthat a separation server 11, a motion detecting server 12, an areaspecifying server 13, a mixture ratio calculating server 14, aforeground/background image separating server 15, a motion bluradjusting server 16, an encoding server 17, storage servers 18-1, 18-2,a synthesizing server 19, a correcting server 20, a purchasing server21, a selling server 22, a retrieving server 23, an account chargeserver 24, a financial transaction server (for customer) 25, a financialtransaction server (for provider) 26, client computers 27 and cameraterminal units 28-1˜28-n are connected over a network 1 such as theInternet to mutually exchange data. The separation server 11, the motiondetecting server 12, the area specifying server 13, the mixture ratiocalculating server 14, the foreground/background image separating server15, the motion blur adjusting server 16, the encoding server 17, thesynthesizing server 19, the correcting server 20, the purchasing server21, the selling server 22, the retrieving server 23, the account chargeserver 24 and the financial transaction servers (for customer and forprovider) 25, 26 are managed or administrated by a provider who providesseparating service, motion detecting service, area specifying service,mixture ratio calculating service, foreground/background separatingservice, motion blur adjusting service, encoding service, synthesizingservice, correcting service, purchasing service, selling service,retrieval service, charging service and financial transaction services(for client and provider), respectively. Further, in the followingdescription, the storage servers 18-1, 18-2 or the camera terminal units28-1 to 28-n are simply named the storage server 18 or the cameraterminal unit 28 when there is no need to distinguish them. Further, itis true of other servers and units.

FIG. 2 is a configuration view of the separation server 11 of thepresent invention.

A CPU (Central Processing Unit) 41 performs various kinds of processingin accordance with programs stored in a ROM (Read Only Memory) 42 or astorage unit 48. Data or programs executed by CPU 41 are stored in a RAM(Random Access Memory) 43 as needed. These CPU 41, ROM 42 and RAM 43 areconnected to one another via a bus 44.

An input/output interface 45 is connected to CPU 41 via bus 44. An inputunit 46 including a keyboard, a mouse, microphone, etc. and an outputunit 47 including a display, speakers, etc. are connected to theinput/output interface 45. The CPU 41 performs various kinds ofprocessing according to instructions input from the input unit 46. TheCPU 41 outputs images, voices, etc. obtained as a result of processingto the output unit 47.

The storage unit 48 connected to the input/output interface 45comprises, for example, a hard disk and stores programs executed by CPU41 or various kinds of data. A communication unit 49 communicates withexternal devices via the Internet or other networks.

Further, programs may be obtained through the communication unit 49 andbe stored in a storage unit 48.

When a magnetic disk 61, an optical disk 62, an optical magnetic disk63, a semiconductor memory, etc. are mounted, a drive 50 connected tothe input/output interface 45 drives them to obtain programs or datastored therein. The obtained programs or data are transferred and storedin the storage unit 48 as needed.

Further, since the basic configuration of the motion detecting server12, the area specifying server 13, the mixture ratio calculating server14, the foreground/background image separating server 15, the motionblur adjusting server 16, the encoding server 17, the storage servers18-1, 18-2, the synthesizing server 19, the correcting server 20, thepurchasing server 21, the selling server 22, the retrieving server 23,the account charge server 24, the financial transaction server (forcustomer) 25, the financial transaction server (for provider) 25 and theclient computers 27 are the same as that of the separation server 11,the description thereof will be omitted.

FIG. 3 is a drawing showing a configuration of the camera terminal unit28 of the present invention. The camera terminal unit 28 is constructedsuch that a sensor 76 a and a GPS (Global Positioning System) 76 b areprovided in an input unit 76, an LCD (Liquid Crystal Display) 77 a isprovided in an output unit 77 and the other structures are the same asthat of the separation server 11. That is, a CPU 71, a ROM 72, a RAM 73,a bus 74, an input/output interface 75, an input unit 76, an output unit77, a storage unit 78, a communication unit 79, a drive 80, a magneticdisk 91, an optical disk 92, an optical magnetic disk 93 and asemiconductor memory 94 of the camera terminal unit 28 correspond to theCPU 41, the ROM 42, the RAM 43, the bus 44, the input/output interface45, the input unit 46, the output unit 47, the storage unit 48, thecommunication unit 49, the drive 50, the magnetic disk 61, the opticaldisk 62, the optical magnetic disk 63 and the semiconductor memory 64 ofthe separation server 11, respectively.

The sensor 76 a is an image pickup element and outputs the picked upimage to the input unit 76. The GPS 76 b detects positional informationon earth (latitude and longitude) based on signals transmitted from astationary satellite (not shown) and outputs the detected positionalinformation to the input unit 76. The LCD 77 a displays the image outputfrom the output unit 77.

Next, referring to FIGS. 4 and 5, the separation server 11 will bedescribed.

As shown in FIG. 4, the separation server 11 separates an image inputfrom, for example, the client computers 27 via the network 1, into aforeground component image and a background component image by means ofthe method to be mentioned later, generates ID for the input image, theforeground component image and the background component image and then,outputs them to the client computers 27, stores them in its own storageunit, outputs them to the storage server 18 to be stored, or outputsthem to other servers on the network to be stored via the network 1.Here, the foreground component image means an image having a movingcomponent of the input image and the background component image means animage of stationary part not having the moving component. At this time,a charge processor 11 a makes the account charge server 24 performcharge processing for the cost of the separation processing via thenetwork 1. Further, as shown in FIG. 5, if an image ID specifying animage is input in place of an image, the separation server 11 accessesthe retrieving server 23 to be described later or the storage server 18on the network 1 or retrieves its own storage unit (for example, thestorage unit 78 of FIG. 3) to read the image data corresponding to theinput image ID and separates the image data into a foreground componentimage and a background component image. Thereafter, the separationserver 11 generates ID for each image, which is stored in its ownstorage unit, or outputs them to other servers on the network 1 so as toperform their respective processing.

Further, in the following description, the image ID is explained as anexample of image specifying information. However, any other informationcapable of specifying an image, for example, positional information ofimage to be described later may be used as the image specifyinginformation.

Next, referring to the FIGS. 6 and 7, the motion detecting server 12will be explained.

As shown in FIG. 6, an object extracting unit 12 a of the motiondetecting server 12 extracts, for example, an image object of imageinput from the client computer 27, etc. and outputs them to a motiondetector 12 b. The motion detector 12 b detects the motion vector andthe positional information of the input image object and then, outputsthem the client computers 27, stores them in its own storage unit oroutputs them to other servers on the network 1 so as to perform theirrespective processing. At this time, a charge processor 12 c makes theaccount charge server 24 perform the charge processing of cost fordetecting the motion vector and the positional information of each imageobject via the network 1. An image that is a target of image pickup andcorresponds to an object in the real world is referred to as an objectin this specification.

Also, as shown in FIG. 7, if an image ID specifying an image is inputinstead of the image, the motion detecting server 12 accesses theretrieving server 23 to be described later or the storage server 18 onthe network 1 or retrieves its own storage unit (for example, thestorage unit 78 of FIG. 3) to read out the image data corresponding tothe input image ID and then, performs the same processing as mentionedabove.

Next, referring to FIGS. 8 and 9, the area specifying server 13 will beexplained.

As shown in FIG. 8, the area specifying server 13 specifies theforeground area, the background area, the mixed area with respect toeach pixel of the input image on the basis of an image input from theclient computer 27 via the network 1, and object specifying informationspecifying object of the input image, generates information(hereinafter, referred to as area information) specifying which pixel isincluded of the foreground area, the background area and the mixed areaand then, outputs it to the client computer 27, stores it in its ownstorage unit or outputs it to other servers on the network 1 so as toperform their respective processing. At this time, a charge processor 13a makes the account charge server 24 perform the charge processing ofcost for the area specification processing via the network 1. Further,as shown in FIG. 9, if an image ID specifying an image is input insteadof the image, the area specifying server 13 accesses the retrievingserver 23 to be described later or a storage server 18 on the network 1,or retrieves its own storage unit (for example, the storage unit 78 ofFIG. 3) to call out the image corresponding to the input image ID andthen, outputs an area information corresponding to object specifyinginformation of the image.

Next, referring to FIGS. 10 and 11, the mixture ratio calculating server14 will be explained.

As shown in FIG. 10, the mixture ratio calculating server 14 calculatesmixture ratios (hereinafter, referred to as mixture ratio α)corresponding to pixels that is included in the mixed area on the basisof an image input from the client computer 27 via the network 1, etc.,the object specifying information for specifying the object of theimage, and the area information and then, outputs them to the clientcomputer 27, stores them in its own storage unit or outputs to otherservers via the network 1 so as to perform their respective processingvia the network 1. At this time, the charge processor 14 a makes theaccount charge server 24 perform the charge processing of cost for themixture ratio calculation processing via the network 1. Further, asshown in FIG. 11, if an image ID specifying an image is input instead ofthe image, the mixture ratio calculating server 14 accesses theretrieving server 23 to be described later or the storage server 18 onthe network 1 or retrieves its own storage unit (for example, thestorage unit 78 by FIG. 3) to call out the image corresponding to theinput image ID and then, performs the same process as mentioned above.

Next, referring to FIGS. 12 and 13, the foreground/background imageseparating server 15 will be explained.

As shown in FIG. 12, the foreground/background image separating server15 separates the input image into a foreground component image havingonly the image components (hereinafter, referred to as foregroundcomponents) corresponding to the foreground objects and a backgroundcomponent image having only the background components (hereinafter,referred to as background components) on the basis of the input imagefrom the client computer 27, etc., the object specifying informationspecifying object of the image, the area information and the mixtureratio α, generates an ID for each image and then, outputs them to theclient computer 27, stores them in its own storage unit, or outputs themto other servers on the network 1 so as to perform their respectiveprocessing via the network 1. At this time, a charge processor 15 amakes the account charge server 24 perform the charge processing of costfor the foreground/background separation processing via the network 1.Further, as shown in FIG. 13, if an image ID specifying the image isinput instead of the image, the foreground/background image separatingserver 15 accesses the retrieving server 23 to be described later or thestorage server 18 on the network 1 or retrieves its own storage unit(for example, the storage unit 78 of FIG. 3) to call out the imagecorresponding to the input image ID and then, performs the sameprocessing as mentioned above.

Next, referring to FIGS. 14 and 15, the motion blur adjusting server 16will be explained.

As shown in FIG. 14, the motion blur adjusting server 16 adjusts theamount of motion blur in the foreground image component by eliminatingthe motion blur in the foreground image component and decreasing orincreasing the amount of motion blur in the foreground component imageon the basis of the foreground component image input from the clientcomputer 27, etc., the motion vector, and the amount of motion blur soas to generate the foreground component image of which the amount ofmotion blur is adjusted, and generates an ID for each image to outputthem to the client computer 27, to store them in its own storage unit orto output them to other server on the network 1 so as to perform theirrespective processing. At this time, the charge processor 16 a makes theaccount charge server 24 perform the charge processing of cost for themotion blur adjusting processing via the network 1. Further, as shown inFIG. 15, if a foreground component image ID specifying the foregroundcomponent image is input instead of the foreground component image, themotion blur adjusting server 16 accesses the retrieving server 23 to bedescribed later or the storage server 18 on the network 1 or retrievesits own storage unit (for example, the storage unit 78 of FIG. 3) tocall out the foreground component image corresponding to the inputforeground components image ID and then, performs the same processing asmentioned above.

Next, referring to FIGS. 16 and 17, the encoding server 17 will beexplained.

As shown in FIG. 16, the encoding server 17 separates an image inputfrom the client computers 27, etc. into a foreground component image anda background component image, generates ID for each image, stores themin its own storage unit or outputs them to other server on the network1, generates positional information of the foreground component imageand positional information of the background component image having acode indicating the position of the server on the network 1, such as URL(Universal Resource Locator) of the server to which the foregroundcomponent image and the background component image were output (stored)and then, outputs them along with information having motion vectors,positional information and mixture ratio for the images via the network1. Further, the information output from the encoding server may be theencoding information, the image and the encoding information, or theimage itself and may be changed as needed. At this time, a chargeprocessor 17 a makes the account charge server 24 perform the chargeprocessing of cost for the encoding processing via the network. Further,as shown in FIG. 17, if an image ID specifying an image is input insteadof an image, the encoding server 17 accesses the retrieving server 23 tobe described later or the storage server 19 on the network 1 orretrieves its own storage unit (for example, the storage unit 78 of FIG.3) to call out the image corresponding to the input image ID, and thenperforms the same processing as mentioned above.

Next, referring to FIGS. 18 and 19, the storage server 18 will beexplained.

As shown in FIG. 18, the storage server 18 is connected via the network1, stores the image transmitted from various kinds of servers, andoutputs an image positional information corresponding to the storedimage along with an image ID. For example, the client computer 27 canaccess via the network 1 and call out the desired image using the imagepositional information. That is, as shown in FIG. 19, for example, theclient computer 27 can access the storage server 18 on the network 1 onthe basis of the image positional information and read out the desiredimage by specifying the image ID corresponding to the desired image.Further, the image positional information and the image ID are explainedseparately in this specification but the image positional informationmay be a part of the image ID. In this case, the image positionalinformation and the server on the network 1 in which the imagepositional information is stored (memorized or processed) can berecognized from the image ID. Further, the storage server 18 may storethe motion vector, the positional information, the mixture ratio and theamount of motion blur as well as the image data.

Next, referring to FIGS. 20 and 21, the synthesizing server 19 will beexplained.

As shown in FIG. 20, the synthesizing server 19 synthesizes images A andB by using two images A, B input from, for example, the client computer27, etc., the motion vector, the positional information, the mixtureratio and the amount of motion blur to generate a synthesized image A+Band then, outputs them to the client computer 27, stores it in its ownstorage unit or outputs it to other servers on the network so as toperform their respective processing via the network 1. In this case, theimages A, B are synthesized by treating one of the images as theforeground component image and the other as the background componentimage. At this time, a charge processor 19 a makes the account chargeserver 24 perform the charge processing of cost for the synthesizingprocess via the network 1. Further, as shown in FIG. 21, if an image AID and an image B ID specifying the image A and B are input instead ofthe images A and B, the synthesizing server 19 accesses the retrievingserver 23 to be described later, the storage server 18 on the network orretrieves its own storage unit (for example, the storage unit 78 of FIG.3) to call out the images corresponding to the input image A ID andimage B ID and then, performs the same processing as mentioned above.

Next, referring to FIGS. 22 and 23, the correcting server 20 will beexplained.

As shown in FIG. 22, the correcting server 20 corrects the image inputfrom, for example, the client computer 27, etc. on the basis of themotion vector, the positional information, the mixture ratio and theamount of motion blur, generates and outputs the corrected image to theclient computer 27, to store its own storage unit or to other servers onthe network 1 so as to perform their respective processing. At thistime, the charge processor 20 a makes the account charge server 24perform the charge processing of cost for the correction processing viathe network 1. Further, as shown in FIG. 23, if an image ID specifyingthe image is input instead of the image, the correcting server 20accesses the retrieving server 23 to be described later or the storageserver 18 or retrieves its own storage unit (for example, the storageunit 78 of FIG. 3) to call out the image corresponding to the inputimage ID and then, performs the same processing as mentioned above.

Next, referring to FIG. 24, the purchasing server 21 will be explained.

As shown in FIG. 24, if an image ID specifying an image desired to bepurchased is input from, for example, the client computer 27, etc. by auser who wants to purchase the image, the purchasing server 21 accessesthe separation server 11, the storage server 18, the synthesizing server19 or the correcting server 20 on the network 1 to call out thecorresponding image and outputs that image to, for example, the clientcomputer 27 via the network. At this time, the charge processor 21 amakes the account charge server 24 perform the charge processing of costfor the purchasing processing via the network 1.

Next, referring to FIG. 25, the selling server 22 will be explained.

As shown in FIG. 25, if an image which may be a separated image, asynthesized image or a corrected image generated, for example, by meansof the separation server 11, the synthesizing server 19 or thecorrecting server 20 is input via the network 1 by a user who wants tosell the image, the selling server 22 stores the image in the separationserver 11, the storage server 18, the synthesizing server 19 or thecorrecting server 20 on the network 1, and the charge processor 22 amakes the account charge server 24 perform the charge processing of costfor the sold image via the network 1 (in this case, the provider ofprocessing of the selling service make the user desiring the saleperform the payment processing of the processing fee corresponding tothe sold image).

Next, referring to FIG. 26, the retrieving server 26 will be explained.The retrieving server 26 retrieves an image which is being picked up orhas been picked up by the camera terminal units 28-1 to 28-n on thenetwork 1 on the basis of information specifying the features of animage desired by a user from the client computer 27 and the physicalpositional information of the camera terminal units 28-1 to 28-n tooutput it to, for example, the client computer 27 as the requestedimage. At this time, the charge processor 23 a makes the account chargeserver 24 perform the charge processing of cost for the retrievalprocess via the network 1.

Further, in this specification, encoding means to convert the image datainto data of the foreground component image, the background componentimage, the motion vector, the positional information, the amount ofmotion blur and the mixture ratio information, and the data thereof isreferred to as encoded data.

FIG. 27 is a block diagram showing the separation server 11.

Further, it does not matter whether each function of the separationserver 11 is implemented by hardware or software. That is, each blockdiagram of the present invention may be considered as a block diagram ofhardware or as a functional block diagram of software.

The input image supplied to the separation server 11 is supplied townobject extracting unit 101, an area specifying unit 103, a mixture ratiocalculator 104 and a foreground/background separator 105.

The object extracting unit 101 roughly extracts an image objectcorresponding to the foreground object of the input image and suppliesthe extracted image object to the motion detector 102. The objectextracting unit 101 detects, for example, the outline of the imageobject corresponding to the foreground object of the input image androughly extracts the image object corresponding to the foregroundobject.

The object extracting unit 101 roughly extracts an image objectcorresponding to the background object of the input image and suppliesthe extracted image object to the motion detector 102. The objectextracting unit 101 roughly extracts the image object corresponding tothe background object on the basis of, for example, the differencebetween the input image and the image object corresponding to theextracted foreground object.

Further, for example, the object extracting unit 101 may roughly extractthe image object corresponding to the foreground object and the imageobject corresponding to the background object on the basis of thedifference between the background image stored in an internally providedbackground memory and the input image.

The motion detector 102 calculates a motion vector for a roughlyextracted image object corresponding to the foreground object by usingtechniques such as a block matching method, an equally-dividing method,a phase correlating method or a pixel recursive method to supply thecalculated motion vector and the positional information of the motionvector (information specifying positions of pixels corresponding to themotion vector) to the area specifying unit 103 and the motion bluradjusting unit 106. The motion vector output from the motion detector102 includes information corresponding to the amount of movement v.

Further, for example, the motion detector 102 may output the motionvector of each image object to the motion blur adjusting unit 106 alongwith the pixel positional information specifying pixels of image object.

The amount of movement v which indicates the positional variation of theimage corresponding to the moving object is expressed in units of pixelinterval. For example, when the object image corresponding to theforeground is moved and displayed at a position that is separated by 4pixels in a frame following a reference frame, the amount of movement vof the object image corresponding to the foreground is 4.

Further, the object extracting unit 101 and the motion detector 102 arerequired for adjusting the amount of motion blur corresponding to themoving object.

The area specifying unit 103 specifies the foreground area, thebackground area or the mixed area for each pixel of the input image andsupplies information indicating the foreground area, the background areaor the mixed area for each pixel, to the mixture ratio calculator 104,the foreground/background separator 105 and the motion blur adjustingunit 106.

The mixture ratio calculator 104 calculates a mixture ratiocorresponding to pixels in the mixed area on the basis of the inputimage and the area information supplied from the area specifying unit103 and supplies the calculated mixture ratio to theforeground/background separator 105.

The mixture ratio α is a value indicating a ratio of image component(hereinafter, referred to as background component) corresponding to thebackground object in pixel values as shown in equation (3) to bedescribed later.

The foreground/background separator 105 separates the input image into aforeground component image having only the image component (hereinafter,referred to as foreground component) corresponding to the foregroundobject and a background component image having only the backgroundcomponent on the basis of the area information supplied from the areaspecifying unit 103 and the mixture ratio α supplied from the mixtureratio calculator 104 to supply the foreground component image to themotion blur adjusting unit 106 and the selector 107. Further, it may beconsidered that the separated foreground component image is the finaloutput. Compared to the conventional methods where only the foregroundand background are specified and separated without considering the mixedarea, more accurate foreground and background can be obtained.

The motion blur adjusting unit 106 determines a unit of processing whichspecifies one or more pixels in the foreground component image, based onthe amount of movement v known from the motion vector and the areainformation. The unit of processing is a data specifying a group ofpixels that is a target of motion blur adjusting process.

The motion blur adjusting unit 106 adjusts the amount of motion blur inthe foreground component image by removing the motion blur in theforeground component image, decreasing the amount of motion blur,increasing the amount of motion blur, etc. based on the amount of motionblur input to the separation server 11, the foreground component imagesupplied from the foreground/background separator 105, the motion vectorand the positional information thereof supplied from the motion detector102 and the unit of processing, and outputs the foreground componentimage of which the amount of motion blur is adjusted to the selector107. The motion vector and the positional information thereof may not beused.

Here, the motion blur means distortion included in an imagecorresponding to a moving object, generated by the motion of object inreal society which is a target of picking up an image and by picking upcharacteristics of sensors.

The selector 107 selects the foreground component image supplied fromthe foreground/background separator 105 or the foreground componentimage supplied from the motion blur adjusting unit 106 of which theamount of motion blur is adjusted, based on, for example, a selectionsignal corresponding to a selection made by a user and then, outputs theselected foreground component image.

Next, with reference to FIGS. 28 through 43, an input image supplied tothe separation server 11 will be described.

FIG. 28 is a diagram illustrating an image taken by a sensor 76 a. Thesensor 76 a comprises, for example, a CCD video camera, etc having a CCD(Charge Coupled Device) area sensor that is a solid state image pickupdevice. An object 112 corresponding to a foreground in a real world ismoved, for example, horizontally from left to right in the drawingbetween the sensor and an object 111 corresponding to a background in areal world.

The sensor 76 a picks up the image of the object 112 corresponding tothe foreground along with the object corresponding to the background.The sensor 76 a outputs the picked up in a frame unit. For example, thesensor 76 a outputs an image having 30 frames per second. The exposuretime of the sensor 76 a may be set to 1/30 second. The exposure timemeans a time interval which starts at the point of converting the inputlight into charges and ends at the point when the input light hasconverted into charges. Hereinafter, the exposure time is referred to asa shutter time.

FIG. 29 is a diagram illustrating an arrangement of pixels. In FIG. 29,‘A’ through ‘I’ indicate the respective pixels. The pixels are arrangedon a plane corresponding to an image. A detection element correspondingto one pixel is arranged on the sensor 76 a. When the sensor 76 a picksup an image, the detection element outputs a pixel value correspondingto one pixel, constituting the image. For example, a position of thedetection element in X direction corresponds to a horizontal position onan image and a position of the detection element in Y directioncorresponds to a vertical position on an image.

As shown in FIG. 30, for example, the CCD detection element converts theinput light into a charge for the time interval corresponding to theshutter time and stores the converted charge. The amount of charge isapproximately proportional to the intensity of input light and timeinterval when light is input. The detection element adds the chargeconverted from the input light to the charge already stored for the timeinterval corresponding to the shutter time. That is, the detectionelement integrates the input light for the time interval correspondingto the shutter time to store the amount of charge corresponding to theintegrated light. The detection element can be mentioned to haveintegration effect with time.

The charge stored in the detection element is converted into a voltagevalue by means of a circuit not shown and the voltage value is convertedinto a pixel value such as digital data, etc. to be output. Therefore,each pixel value output from the sensor 76 a has a value projected toone-dimensional space as a result of integrating some portion havingspatial extension of an object corresponding to foreground or backgroundwith respect to the shutter time.

The separation server 11 extracts any significant information, forexample, mixture ratio α, hidden in the output signal by means ofstoring operation of such sensor 76 a. The separation server 11 adjuststhe amount of distortion, for example, the amount of motion blurresulting from the mixture of the same image object of the foreground.Further, the separation server 11 adjusts the amount of distortionresulting from the mixture of image object of the foreground and imageobject of the background.

FIG. 31A through 31D are diagrams showing an image obtained by takingpicture of an object corresponding to a moving foreground and an objectcorresponding to a stationary background. FIG. 31A shows an imageobtained by taking picture of an object corresponding to a movingforeground and an object corresponding to a stationary background. Inthe example shown in FIG. 31A, the object corresponding to theforeground moves horizontally from left to right with respect to thescreen.

FIG. 31B is a model diagram in which the pixel values corresponding toone line of the image shown in FIG. 31A are expanded in the timedirection. The horizontal direction in FIG. 31B corresponds to thespatial direction X in FIG. 31A.

The pixel values of pixels in the background area comprise only thecomponents of the background, that is, the components of the imagecorresponding to the background object. The pixel values of pixels inthe foreground area comprise only the components of the foreground, thatis, the components of the image corresponding to the foreground object.

The pixel values of pixels in the mixed area comprise the components ofthe background and the components of the foreground. Since the pixelvalues of pixels in the mixed area comprise the components of thebackground and the components of the foreground, the mixed area can bementioned as a distortion area. Further, the mixed area is classifiedinto a covered background area and an uncovered background area.

The covered background area is a mixed area corresponding to a front endportion in a direction in which the foreground object moves in respectto the foreground area and is an area in which the background componentsis hidden behind the foreground with time elapsing.

In contrast, the uncovered background area is a mixed area correspondingto the rear end portion in a direction in which the foreground objectmoves in respect to the foreground area and is an area in which thebackground components appear with time elapsing.

As such, the image including the foreground area, the background area,the covered background area or the uncovered background area is input tothe area specifying unit 103, the mixture ratio calculator 104 and theforeground/background separator 105 as the input image.

FIG. 32 is a diagram illustrating the background area, the foregroundarea, the mixed area, the covered background area, and the uncoveredbackground area like above. When an image corresponds to the image shownin FIGS. 31A and 31B, the background area is a stationary portion, theforeground area is a moving portion, the covered background area of themixed area is a portion being changed from the background to theforeground and the uncovered background area of the mixed area is apotion being changed from the foreground to the background.

FIG. 33 is a model diagram in which the pixel values of pixels beingadjacent in a row are expanded in the time direction in the imageobtained by picking up an image of an object corresponding to thestationary foreground and an object corresponding to the stationarybackground. For example, as pixels being adjacent in a row, pixels beingadjacent to a line of screen can be selected.

Pixel values of F01 to F04 shown in FIG. 33 are pixel values of pixelscorresponding to a stationary foreground object. Pixel values of B01 toB04 shown in FIG. 33 are pixel values of pixels corresponding to astationary background object.

The vertical direction in FIG. 33 corresponds to time and time elapsesfrom the upper side to the lower side in the drawing. The Position atthe upper side of the rectangle in FIG. 33 corresponds to a time whenthe sensor 76 a starts to convert of the input light to charges and theposition at the lower side of the rectangle in FIG. 33 corresponds totime when the sensor 76 a finishes converting the input light intocharges. That is, the distance from the upper side to the lower side ofthe rectangle in FIG. 33 corresponds to a shutter time.

Now, a case of when the shutter time equals to the frame interval willbe explained as an example.

The horizontal direction in FIG. 33 corresponds to the spatial directionx shown in FIG. 31A. More specifically, in the example shown in FIG. 33,the distance from the left side of rectangle indicated by “F01” to theright side of rectangle indicated by “B04” in FIG. 33 corresponds to 8times of a pixel pitch, that is, intervals of 8 pixels being continuous.

When the object of foreground and the object of background arestationary, light to be input to the sensor 76 a is not varied for atime interval corresponding to a shutter time.

Here, the time interval corresponding to the shutter time is dividedinto two or more equivalent lengths. For example, if a virtual divisionnumber is 4, the model diagram shown in FIG. 33 can be shown as themodel diagram shown in FIG. 9. The virtual division number isestablished according to the amount of movement v for the shutter timeof the object corresponding to the foreground. For example, if theamount of movement v is defined as 4, the virtual division number is setat 4 and the time interval corresponding to the shutter time is dividedinto 4 intervals.

The uppermost row in the drawing corresponds to a first divided timeinterval after the shutter is opened. The second row from the uppermostin the drawing corresponds to a second divided time interval after theshutter is opened. The third row from the uppermost in the drawingcorresponds to a third divided time interval after the shutter isopened. The fourth row from the uppermost in the drawing corresponds toa fourth divided time interval after the shutter is opened.

Now, the time interval by which the shutter time is divided according tothe amount of movement v is referred to as shutter time/v.

When the object corresponding to the foreground is stationary, sincelight input to the sensor 76 a is not varied, the foreground componentF01/v has the same value the pixel value F01 divided by the virtualdivision number. Similarly, when the object corresponding to theforeground is stationary, the foreground component F02/v has the samevalue as the pixel value F02 divided by the virtual division number, theforeground component F03/v has the same value as the pixel value F03divided by the virtual division number and the foreground componentF04/v has the same value as the pixel value F04 divided by the virtualdivision number.

When the object corresponding to the background is stationary, sincelight input to the sensor 76 a is not varied, the background componentB01/v has the same value as the pixel value B01 divided by the virtualdivision number. Similarly, when the object corresponding to thebackground is stationary, the background component B02/v has the samevalue as the pixel value B02 divided by the virtual division number, thebackground component B03/v has the same value as the pixel value B03divided by the virtual division number and the background componentB04/v has the same value as the pixel value B04 divided by the virtualdivision number.

That is, when an object corresponding to the foreground is stationary,since light input to the sensor 76 a is not varied for the time intervalcorresponding to the shutter time, the foreground component F01/vcorresponding to a first shutter time/v after the shutter is opened, theforeground component F01/v corresponding to a second shutter time/vafter the shutter is opened, the foreground component F01/vcorresponding to a third shutter time/v after the shutter is opened andthe foreground component F01/v corresponding to a fourth shutter time/vafter the shutter is opened all have the same value. F02/v to F04/v havealso the same relation as F01/v.

When an object corresponding to the background is stationary, sincelight input to the sensor 76 a is not varied for the time intervalcorresponding to the shutter time, the background component B01/vcorresponding to a first shutter time/v after the shutter is opened, thebackground component B01/v corresponding to a second shutter time/vafter the shutter is opened, the background component B01/vcorresponding to a third shutter time/v after the shutter is opened andthe background component B01/v corresponding to a fourth shutter time/vafter the shutter is opened all have the same value. B02/v to B04/v havethe same value as B01/v.

Next, a case of when an object corresponding to the foreground is movingand the object corresponding to the background is stationary will beexplained.

FIG. 35 is a model diagram in which when an object corresponding to theforeground is moving toward the right side in the drawing, the pixelvalues of pixels in a line including the covered background area aredeveloped in the time direction. In FIG. 35, the amount of movement v ofthe foreground is 4. Since one frame corresponds to a short time, it canbe supposed that the object corresponding to the foreground is a rigidbody and is moving at a uniform speed. In FIG. 35, an image of theobject corresponding to the foreground is moving to be displayed atposition displaced by 4 pixels in a frame following a reference frame.

In FIG. 35, the pixels from the leftmost to the fourth from the leftmostbelong to the foreground. In FIG. 35, the pixels from the fifth from theleftmost to the seventh from the leftmost belong to the mixed area thatis the covered background area. In FIG. 35, the rightmost pixel belongsto the background area.

Since the object corresponding to the foreground is moving to cover theobject corresponding to the background with time elapsing, thecomponents included in the pixel values of pixels belonging to thecovered background area is changed from the background components to theforeground components at certain time point of the time intervalcorresponding to the shutter time.

For example, the pixel values M of which edge is a bold line in FIG. 35is expressed as equation (1).

M=B02/v+B02/v+F07/v+F06/v   (1)

For example, since the fifth pixel from the leftmost includes thebackground component corresponding to a first shutter time/v and theforeground component corresponding to a third shutter time/v, themixture ratio α of the fifth pixel from the leftmost is ¼. Since thesixth pixel from the leftmost includes the background componentcorresponding to a second shutter time/v and the foreground componentcorresponding to the second shutter time/v, the mixture ratio α of thesixth pixel from the leftmost is ½. Since the seventh pixel from theleftmost includes the background component corresponding to the thirdshutter time/v and the foreground component corresponding to the firstshutter time/v, the mixture ratio α of the seventh pixel from theleftmost is ¾.

Since it can be supposed that the object corresponding to the foregroundis a rigid body and the foreground image is moving at an uniform speedto be displayed at a position displaced to the right by 4 pixels in anext frame, for example, the foreground component F07/v, of the fourthpixel from the leftmost in FIG. 35, corresponding to the first shuttertime/v after the shutter is opened is the same as the foregroundcomponent of the fifth pixel from the leftmost in FIG. 35, correspondingto the second shutter time/v after the shutter is opened. Similarly, theforeground component F07/v is the same as the foreground component ofthe sixth pixel from the leftmost in FIG. 35 corresponding to the thirdshutter time/v after the shutter is opened and the foreground componentof the seventh pixel from the leftmost in FIG. 35 corresponding to thefourth shutter time/v after the shutter is opened, respectively.

Since it can be supposed that the object corresponding to the foregroundis a rigid body and the foreground image is moving at an uniform speedto be displayed at a position displaced to the right by 4 pixels in anext frame, for example, the foreground component F06/v of the thirdpixel from the leftmost in FIG. 35 corresponding to the first shuttertime/v after the shutter is opened is the same as the foregroundcomponent of the fourth pixel from the leftmost in FIG. 35 correspondingto the second shutter time/v after the shutter is opened. Similarly, theforeground component F06/v is the same as the foreground component ofthe fifth pixel from the leftmost in FIG. 35 corresponding to the thirdshutter time/v after the shutter is opened and the foreground componentof the sixth pixel from the leftmost in FIG. 35 corresponding to thefourth shutter time/v after the shutter is opened, respectively.

Since it can be supposed that the object corresponding to the foregroundis a rigid body and the foreground image is moving at an uniform speedto be displayed at a position displaced to the right by 4 pixels in anext frame, for example, the foreground component F05/v of the secondpixel from the leftmost in FIG. 35 corresponding to the first shuttertime/v after the shutter is opened is the same as the foregroundcomponent of the third pixel from the leftmost in FIG. 35 correspondingto the second shutter time/v after the shutter is opened. Similarly, theforeground component F05/v is the same as the foreground component ofthe fourth pixel from the leftmost in FIG. 35 corresponding to the thirdshutter time/v after the shutter is opened and the foreground componentof the fifth pixel from the leftmost in FIG. 35 corresponding to thefourth shutter time/v after the shutter is opened, respectively.

Since it can be supposed that the object corresponding to the foregroundis a rigid body and the foreground image is moving at an uniform speedto be displayed at a position displaced to the right by 4 pixels in anext frame, for example, the foreground component F04/v of the leftmostpixel in FIG. 35 corresponding to the first shutter time/v after theshutter is opened is the same as the foreground component of the secondpixel from the leftmost in FIG. 35 corresponding to the second shuttertime/v after the shutter is opened. Similarly, the foreground componentF04/v is the same as the foreground component of the third pixel fromthe leftmost in FIG. 35 corresponding to the third shutter time/v afterthe shutter is opened and the foreground component of the-fourth pixelfrom the leftmost in FIG. 35 corresponding to the fourth shutter time/vafter the shutter is opened, respectively.

The foreground area corresponding to a moving object may be mentioned asa distortion area because it includes the motion blur as described bythe above.

FIG. 36 is a model diagram in which when the foreground is moving towardthe right side in the drawing, the pixel values of pixels in a lineincluding the uncovered background area are expanded in the timedirection. In FIG. 36, the amount of movement v of the foreground is 4.Since one frame corresponds to a short time, it can be supposed that anobject corresponding to the foreground is a rigid body and is moving ata uniform speed. In FIG. 36, an image of the object corresponding to theforeground is moving toward the right side by 4 pixels in a next frame.

In FIG. 36, pixels from the leftmost to the fourth from the leftmostbelong to the background area. In. FIG. 36, pixels from the fifth fromthe leftmost to the seventh from the leftmost belong to the mixed areathat is the uncovered background area. In FIG. 36, the rightmost pixelbelongs to the background area.

Since the object corresponding to the foreground covering the objectcorresponding to the background is moving to be displaced from theobject corresponding to the background with time elapsing, thecomponents included in the pixel values of pixels belonging to theuncovered background area is changed from the foreground components tothe background components at certain time point of the time intervalcorresponding to the shutter time.

For example, the pixel values M′ of which edge is a bold line in FIG. 36is expressed as equation (2).

M′=F02/v+F01/v+B26/v+B26/v   (2)

For example, since the fifth pixel from the leftmost includes thebackground component corresponding to a third shutter time/v and theforeground component corresponding to a first shutter time/v, themixture ratio α of the fifth pixel from the leftmost is ¾. Since thesixth pixel from the leftmost includes the background componentcorresponding to a second shutter time/v and the foreground componentcorresponding to the second shutter time/v, the mixture ratio α of thesixth pixel from the leftmost is ½. Since the seventh pixel from theleftmost includes the background component corresponding to the firstshutter time/v and the foreground component corresponding to the thirdshutter time/v, the mixture ratio α of the seventh pixel from theleftmost is ¼.

More generalizing combining equation (1) and equation (2), the pixelvalue M can also be expressed by equation (3).

$\begin{matrix}{M = {{\alpha \cdot B} + {\sum\limits_{i}{F_{i}/v}}}} & (3)\end{matrix}$

Here, α is a mixture ratio. B is a pixel value of the background andFi/v is a foreground component.

Since it can be supposed that the object corresponding to the foregroundis a rigid body and is moving at an uniform speed and the amount ofmovement v is 4, for example, the foreground component F01/v of thefifth pixel from the leftmost in FIG. 36 corresponding to the firstshutter time/v after the shutter is opened is the same as the foregroundcomponent of the sixth pixel from the leftmost in FIG. 36 correspondingto the second shutter time/v after the shutter is opened. Similarly, theforeground component F01/v is the same as the foreground component ofthe seventh pixel from the leftmost in FIG. 36 corresponding to thethird shutter time/v after the shutter is opened and the foregroundcomponent of the eighth pixel from the leftmost in FIG. 36 correspondingto the fourth shutter time/v after the shutter is opened, respectively.

Since it can be supposed that the object corresponding to the foregroundis a rigid body and is moving at an uniform speed and the virtualdivision number is 4, for example, the foreground component F02/v of thesixth pixel from the leftmost in FIG. 36 corresponding to the firstshutter time/v after the shutter is opened is the same as the foregroundcomponent of the seventh pixel from the leftmost in FIG. 36corresponding to the second shutter time/v after the shutter is opened.Similarly, the foreground component F02/v is the same as the foregroundcomponent of the eighth pixel from the leftmost in FIG. 36 correspondingto the third shutter time/v after the shutter is opened.

Since it can be supposed that the object corresponding to the foregroundis a rigid body and is moving at an uniform speed and the amount ofmovement v is 4, for example, the foreground component F03/v of theseventh pixel from the leftmost in FIG. 36 corresponding to the firstshutter time/v after the shutter is opened is the same as the foregroundcomponent of the eighth pixel from the leftmost in FIG. 36 correspondingto the second shutter time/v after the shutter is opened.

In the explanations of FIGS. 34 to 36, it has been explained that thevirtual division number is 4, however the virtual division numbercorresponds to the amount of movement v. The amount of movement vusually corresponds to the moving speed of an object corresponding tothe foreground. For example, when an object corresponding to theforeground is moving to be displayed at position 4 pixels to the rightin a frame following a reference frame, the amount of movement v isdefined as 4. In accordance to the amount of movement v, the virtualdivision number is set at 4. Similarly, for example, when an objectcorresponding to the foreground is moving to be displayed at a position6 pixels to the right in a frame following a reference frame, the amountof movement v is defined as 6 and the virtual division number is set at6.

FIGS. 37 and 38 show a relationship between the foreground area, thebackground area, the mixed area having the covered background area andthe uncovered background area and the foreground component or thebackground component corresponding to the divided shutter time.

FIG. 37 shows an example in which the pixels of the foreground area, thebackground area and the mixed area are extracted from an image includingthe foreground corresponding to the object moving in front of thestationary background. In an example shown in FIG. 37, an object Acorresponding to the foreground is moving horizontally with respect tothe screen.

A frame #n+1 is a frame next to a frame #n and a frame #n+2 is a framenext to the frame #n+1.

A model in which the pixels of the foreground area, the background areaand the mixed area extracted from one of the frame #n to the frame #n+2and the pixel values of the extracted pixels are expanded in the timedirection with the amount of movement v being 4 is shown in FIG. 38.

Since the object A corresponding to the foreground is moving, the pixelvalues of the foreground area comprise 4 different foreground componentsaccording to intervals of a shutter time/v. For example, the leftmostpixel in the foreground area shown in FIG. 38 comprises F01/v, F02/v,F03/v and F04/v. That is, the pixels of the foreground area include themotion blur.

Since an object corresponding to the background is stationary, the lightcorresponding to the background input to the sensor 76 a for the timeinterval corresponding to the shutter time does not change. In thiscase, the pixel values of the background area do not include the motionblur.

The pixel values of pixels belonging to the mixed area containing thecovered background area or the uncovered background area is comprisedthe foreground components and the background components.

Next, a model will be explained that when an image corresponding to anobject is moving, the pixel values of pixels are adjacent in one row ina plurality of frames and are at the same positions in the frames areexpanded in the time direction. For example, when the imagecorresponding to the object is moving horizontally with respect to thescreen, the pixels being in one line on the screen can be selected aspixels being adjacent in one row.

FIG. 39 is a model diagram in which the pixel values of pixels areexpanded in the time direction, wherein the pixel values of pixels whichare adjacent in one row in 3 frames of an image obtained by picking upan image of an object corresponding to the stationary background and areat the same positions in the frames. A frame #n is a frame following aframe #n−1 and a frame #n+1 is a frame following the frame #n. Otherframes are referred in the same manner.

The pixel values of B01 to B12 shown in FIG. 39 are the pixel valuescorresponding to the object of the stationary background. Since theobject corresponding to the background is stationary, the pixel valuesof the corresponding pixels do not changed in the frame #n−1 to theframe # n+1. For example, a pixel in the frame #n and a pixel in theframe #n+1 corresponding to the position of pixel having the pixel valueof B05 in the frame #n−1 have the same pixel value of B05, respectively.

FIG. 40 is a model diagram in which the pixel values of pixels areexpanded in the time direction, wherein the pixel values of pixels areadjacent in one row in 3 frames of an image obtained by picking up animage of an object corresponding to the foreground moving toward theright side in the drawing along with an object corresponding to thestationary background and are at the same positions in the frames. Themodel shown in FIG. 40 includes the covered background area.

In FIG. 40, since it can be supposed that the object corresponding tothe foreground is a rigid body and is moving at an uniform speed and theforeground image is moving to be displayed displaced to the right by 4pixels in a next frame, the amount of movement v of the foreground isdefined as 4 and the virtual division number is set at 4.

For example, the foreground component of the leftmost pixel of a frame#n−1 in FIG. 40 corresponding to a first shutter time/v after theshutter is opened is F12/v, and the foreground component of the secondpixel from the leftmost in FIG. 40 corresponding to a second shuttertime/v after the shutter is opened is also F12/v. Similarly, theforeground component of the third pixel from the leftmost in FIG. 40corresponding to a third shutter time/v after the shutter is opened andthe foreground component of the fourth pixel from the leftmost in FIG.40 corresponding to a fourth shutter time/v after the shutter is openedare F12/v.

The foreground component of the leftmost pixel of the frame #n−1 in FIG.40 corresponding to the second shutter time/v after the shutter isopened is F11/v, and the foreground component of the second pixel fromthe leftmost in FIG. 40 corresponding to the third shutter time/v afterthe shutter is opened is also F11/v. The foreground component of thethird pixel from the leftmost in FIG. 40 corresponding to the fourthshutter time/v after the shutter is opened is F11/v.

The foreground component of the leftmost pixel of the frame #n−1 in FIG.40 corresponding to the third shutter time/v after the shutter is openedis F10/v, and the foreground component, of the second pixel from theleftmost in FIG. 40 corresponding to the fourth shutter time/v after theshutter is opened is also F10/v. The foreground component of theleftmost pixel of the frame #n−1 in FIG. 40 corresponding to the fourthshutter time/v after the shutter is opened is F09/v.

Since an object corresponding to the background is stationary, thebackground component of the second pixel from the leftmost of the frame#n−1 in FIG. 40 corresponding to the first shutter time/v after theshutter is opened is B01/v. The background components of the third pixelfrom the leftmost of the frame #n−1 in FIG. 40 corresponding to thefirst and the second shutter time/v after the shutter is opened areB02/v. The background components of the fourth pixel from the leftmostof the frame #n−1 in FIG. 40 corresponding to the first to third shuttertime/v after the shutter is opened are B03/v.

In the frame #n−1 of FIG. 40, the leftmost pixel corresponds to theforeground area and the second to fourth pixels from the leftmost belongto the mixed area that is the covered background area.

The fifth to twelfth pixels from the leftmost of the frame #n−1 in FIG.40 belong to the background area and the pixel values thereof are B04 toB11, respectively.

The first to fifth pixels from the leftmost of a frame #n in FIG. 40belong to the foreground area. In the foreground area of the frame #n,the foreground component of the shutter time/v is one of F05/v to F12/v.

Since it can be supposed that the object corresponding to the foregroundis a rigid body and is moving at an uniform speed and the foregroundimage is moving to be displayed displaced to the right by 4 pixels in anext frame, the foreground component of the fifth pixel from theleftmost of the frame #n in FIG. 40 corresponding to the first shuttertime/v after the shutter is opened is F12/v, and the foregroundcomponent of the sixth pixel from the leftmost in FIG. 40 correspondingto the second shutter time/v after the shutter is opened is also F12/v.Similarly, the foreground component of the seventh pixel from theleftmost in FIG. 40 corresponding to the third shutter time/v after theshutter is opened and the foreground component of the eighth pixel fromthe leftmost in FIG. 40 corresponding to the fourth shutter time/v afterthe shutter is opened are F12/v.

The foreground component of the fifth pixel from the leftmost of theframe #n in FIG. 40 corresponding to the second shutter time/v after theshutter is opened is F11/v, and the foreground component of the sixthpixel from the leftmost in FIG. 40 corresponding to the third shuttertime/v after the shutter is opened is also F11/v. The foregroundcomponent of the seventh pixel from the leftmost in FIG. 40corresponding to the fourth shutter time/v after the shutter is openedis F11/v.

The foreground component of the fifth pixel from the leftmost of theframe #n in FIG. 40 corresponding to the ‘third shutter time/v after theshutter is opened is F10/v, and the foreground component of the sixthpixel from the leftmost in FIG. 40 corresponding to the fourth shuttertime/v after the shutter is opened is also F10/v. The foregroundcomponent of the fifth pixel from the leftmost of the frame #n in FIG.40 corresponding to the fourth shutter time/v after the shutter isopened is F09/v.

Since the object corresponding to the background is stationary, thebackground component of the sixth pixel from the leftmost of the frame#n in FIG. 40 corresponding to the first shutter time/v after theshutter is opened is B05/v. The background components of the seventhpixel from the leftmost of the frame #n in FIG. 40 corresponding to thefirst and the second shutter time/v after the shutter is opened areB06/v. The background components of the eighth pixel from the leftmostof the frame #n in FIG. 40 corresponding to the first to third shuttertime/v after the shutter is opened are B07/v.

In the frame #n in FIG. 40, the sixth to eighth pixels from the leftmostbelong to the mixed area that is the covered background area.

The ninth to twelfth pixels from the leftmost of the frame #n in FIG. 40belong to the background area and the pixel values thereof are B08 toB11, respectively.

The first to ninth pixels from the leftmost of a frame #n+1 in FIG. 40belong to the foreground area. In the foreground area of the frame #n+1,the foreground component corresponding to the shutter time/v is one ofF01/v to F12/v.

Since it can be supposed that the object corresponding to the foregroundis a rigid body and is moving at an uniform speed and the foregroundimage is moving to be displayed displaced to the right by 4 pixels in anext frame, the foreground component of the ninth pixel from theleftmost of the frame #n+1 in FIG. 40 corresponding to the first shuttertime/v after the shutter is opened is F12/v, and the foregroundcomponent of the tenth pixel from the leftmost in FIG. 40 correspondingto the second shutter time/v after the shutter is opened is also F12/v.The foreground component of the eleventh pixel from the leftmost in FIG.40 corresponding to the third shutter time/v after the shutter is openedand the foreground component of the twelfth pixel from the leftmost inFIG. 40 corresponding to the fourth shutter time/v after the shutter isopened are F12/v.

The foreground component of the ninth pixel from the leftmost of theframe #n+1 in FIG. 40 corresponding to the second shutter time/v afterthe shutter is opened is F11/v, and the foreground component of thetenth pixel from the leftmost in FIG. 40 corresponding to the thirdshutter time/v after the shutter is opened is also F11/v. The foregroundcomponent, of the eleventh pixel from the leftmost in FIG. 40,corresponding to the fourth shutter time/v after the shutter is openedis F11/v.

The foreground component of the ninth pixel from the leftmost of theframe #n+1 in FIG. 40 corresponding to the third shutter time/v afterthe shutter is opened is F10/v, and the foreground component of thetenth pixel from the leftmost in FIG. 40 corresponding to the fourthshutter time/v after the shutter is opened is also F10/v. The foregroundcomponent of the ninth pixel from the leftmost of the frame #n+1 in FIG.40 corresponding to the fourth shutter time/v after the shutter isopened is F09/v.

Since the object corresponding to the background is stationary, thebackground component of the tenth pixel from the leftmost of the frame#n+1 in FIG. 40 corresponding to the first shutter time/v after theshutter is opened is B09/v. The background components of the eleventhpixel from the leftmost of the frame #n+1 in FIG. 40 corresponding tothe first and the second shutter time/v after the shutter is opened areB10/v. The background components of the twelfth pixel from the leftmostof the frame #n+1 in FIG. 40 corresponding to the first to third shuttertime/v after the shutter is opened are B11/v.

In the frame #n+1 of FIG. 40, the tenth to twelfth pixels from theleftmost belong to the mixed area that is the covered background area.

FIG. 41 is a model diagram of an image obtained by extracting theforeground components from the pixel values shown in FIG. 40.

FIG. 42 is a model diagram in which the pixel values of pixels areexpanded in the time direction, wherein the pixel values of pixels areadjacent in one row in 3 frames of an image obtained by picking up animage of an object corresponding to the foreground moving toward theright side in the drawing along with the stationary background and areat the same positions in the frames. In FIG. 42, the uncoveredbackground area is included.

In FIG. 42, it can be supposed that the object corresponding to theforeground is a rigid body and is moving at a uniform speed. Since theobject corresponding to the foreground is moving to be displayeddisplaced to the right by 4 pixels in a next frame, the amount ofmovement v of the foreground is defined as 4.

For example, the foreground component of the leftmost pixel of a frame#n−1 in FIG. 42 corresponding to a first shutter time/v after theshutter is opened is F13/v, and the foreground component of the secondpixel from the leftmost in FIG. 42 corresponding to a second shuttertime/v after the shutter is opened is also F13/v. The foregroundcomponent of the third pixel from the leftmost in FIG. 42 correspondingto a third shutter time/v after the shutter is opened and the foregroundcomponent of the fourth pixel from the leftmost in FIG. 42 correspondingto a fourth shutter time/v after the shutter is opened are F13/v.

The foreground component of the second pixel from the leftmost of theframe #n−1 in FIG. 42 corresponding to the first shutter time/v afterthe shutter is opened is F14/v, and the foreground component of thethird pixel from the leftmost in FIG. 42 corresponding to the secondshutter time/v after the shutter is opened is also F14/v. The foregroundcomponent of the third pixel from the leftmost in FIG. 42 correspondingto the first shutter time/v after the shutter is opened is F15/v.

Since an object corresponding to the background is stationary, thebackground components of the leftmost pixel of the frame #n−1 in FIG. 42corresponding to the second to fourth shutter time/v after the shutteris opened are B25/v. The background components of the second pixel fromthe leftmost of the frame #n−1 in FIG. 42 corresponding to the third andfourth shutter time/v after the shutter is opened are B26/v. Thebackground component of the third pixel from the leftmost of the frame#n−1 in FIG. 42 corresponding to the fourth shutter time/v after theshutter is opened is B27/v.

In the frame #n−1 of FIG. 42, the first, second and third pixels fromthe leftmost belong to the mixed area that is the uncovered backgroundarea.

The fourth to twelfth pixels from the leftmost in the frame #n−1 in FIG.42 belong to the foreground. The foreground components of the frame areone of F13/v to F24/v.

The first, second, third and fourth pixels from the leftmost in theframe #n in FIG. 42 belong to the background area and the pixel valuesthereof are B25 to B28, respectively.

Since it can be supposed that the object corresponding to the foregroundis a rigid body and is moving at an uniform speed and the foregroundimage is moving to be displayed displaced to the right by 4 pixels in anext frame, the foreground component of the fifth pixel from theleftmost of the frame #n in FIG. 42 corresponding to the first shuttertime/v after the shutter is opened is F13/v, and the foregroundcomponent of the sixth pixel from the leftmost in FIG. 42 correspondingto the second shutter time/v after the shutter is opened is also F13/v.The foreground component of the seventh pixel from the leftmost in FIG.42 corresponding to the third shutter time/v after the shutter is openedand the foreground component of the eighth pixel from the leftmost inFIG. 42 corresponding to the fourth shutter time/v after the shutter isopened are F13/v.

The foreground component of the sixth pixel from the leftmost of theframe #n in FIG. 42 corresponding to the first shutter time/v after theshutter is opened is F14/v, and the foreground component of the seventhpixel from the leftmost in FIG. 42 corresponding to the second shuttertime/v after the shutter is opened is also F14/v. The foregroundcomponent of the eighth pixel from the leftmost in FIG. 42 correspondingto the first shutter time/v after the shutter is opened is F15/v.

Since the object corresponding to the background is stationary, thebackground components of the fifth pixel from the leftmost of the frame#n in FIG. 42 corresponding to the second to the fourth shutter time/vafter the shutter is opened are B29/v. The background components of thesixth pixel from the leftmost of the frame #n in FIG. 42 correspondingto the third and the fourth shutter time/v after the shutter is openedare B30/v. The background components of the seventh pixel from theleftmost of the frame #n in FIG. 42 corresponding to the fourth shuttertime/v after the shutter is opened are B31/v.

In the frame #n of FIG. 42, the fifth to seventh pixels from theleftmost belong to the mixed area that is the uncovered background area.

The eighth to twelfth pixels from the leftmost of the frame #n in FIG.42 belong to the foreground area. In the foreground area of the frame#n, the values corresponding to the shutter time/v are one of F13/v toF20/v.

The leftmost to eighth pixels from the leftmost of a frame #n+1 in FIG.42 belong to the background area, and the pixel values thereof are B25to B32, respectively.

Since it can be supposed that the object corresponding to the foregroundis a rigid body and is moving at an uniform speed and the foregroundimage is moving to be displayed displaced to the right by 4 pixels in anext frame, the foreground component of the ninth pixel from theleftmost of the frame #n+1 in FIG. 42 corresponding to the first shuttertime/v after the shutter is opened is F13/v, and the foregroundcomponent of the tenth pixel from the leftmost in FIG. 42 correspondingto a second shutter time/v after the shutter is opened is also F13/v.The foreground component of the eleventh pixel from the leftmost in FIG.42 corresponding to the third shutter time/v after the shutter is openedand the foreground component of the twelfth pixel from the leftmost inFIG. 42 corresponding to the fourth shutter time/v after the shutter isopened are F13/v.

The foreground component of the tenth pixel from the leftmost of theframe #n+1 in FIG. 42 corresponding to the first shutter time/v afterthe shutter is opened is F14/v, and the foreground component of theeleventh pixel from the leftmost in FIG. 42 corresponding to the secondshutter time/v after the shutter is opened is also F14/v. The foregroundcomponent of the twelfth pixel from the leftmost in FIG. 42,corresponding to the first shutter time/v after the shutter is opened isF15/v.

Since the object corresponding to the background is stationary, thebackground components of the ninth pixel from the leftmost of the frame#n+1 in FIG. 42 corresponding to the second to the fourth shutter time/vafter the shutter is opened are B33/v. The background components of thetenth pixel from the leftmost of the frame #n+1 in FIG. 42 correspondingto the third and the fourth shutter time/v after the shutter is openedare B34/v. The background component of the eleventh pixel from theleftmost of the frame #n+1 in FIG. 42 corresponding to the fourthshutter time/v after the shutter is opened is B35/v.

In the frame #n+1 of FIG. 42, the ninth to eleventh pixels from theleftmost belong to the mixed area that is the uncovered background area.

The twelfth pixel from the leftmost of the frame #n+1 in FIG. 42 belongsto the foreground. In the foreground area of the frame #n+1, theforeground components corresponding to the shutter time/v are one ofF13/v to F16/v.

FIG. 43 is a model diagram of an image obtained by extracting theforeground components from the pixel values shown in FIG. 42.

Returning to FIG. 27, the area specifying unit 103 generates flagsindicating which of the foreground area, the background area, thecovered background area or the uncovered background area correspond tothe respective pixels, using a plurality of pixel values, and then,supplies them to the mixture ratio calculator 104 and the motion bluradjusting unit 106 as the area information.

The mixture ratio calculator 104 calculates the mixture ratios a foreach pixel included in the mixed area on the basis of the pixel valuesfrom a plurality of frames and the area information, and supplies thecalculated mixture ratio α to the foreground/background separator 105.

The foreground/background separator 105 extracts the foregroundcomponent image having only the foreground components on the basis ofthe pixel values from a plurality of frames, the area information andthe mixture ratio α and supplies it to the motion blur adjusting unit106.

The motion blur adjusting unit 106 adjusts the amount of motion blurincluded in the foreground component image on the basis of theforeground component image supplied from the foreground/backgroundseparator 105, the motion vector supplied from the motion detector 102and the area information supplied from the area specifying unit 103 andthen outputs the foreground component image in which the motion blur isadjusted.

Referring to the flowchart of FIG. 44, the processing of adjusting theamount of motion blur by means of the separation server 11 will beexplained. In the step 511, the area specifying unit 103 performs thearea specifying processing for generating the area informationindicating to which of the foreground area, the background area, thecovered background area and the uncovered background area that eachpixel of the input image belongs on the basis of the input image.Details of the area specifying processing will be explained later. Thearea specifying unit 103 supplies the generated area information to themixture ratio calculator 104.

Further, in step S11, the area specifying unit 103 may generate the areainformation indicating to which of the foreground area, the backgroundarea and the mixed area (the covered background area and the uncoveredbackground area are not distinguished) that each pixel of the inputimage belongs on the basis of the input image. In this case, theforeground/background separator 105 and the motion blur adjusting unit106 determine whether the mixed area is the covered background area orthe uncovered background area on the basis of the direction of themotion vector. For example, when the order of the foreground area, thebackground area and the mixed area are sequentially arranged for thedirection of the motion vector, it is determined that the mixed area isthe covered background area. when the order of the background area, themixed area and the foreground area are sequentially arranged in thedirection of the motion vector, it is determined that the mixed area isthe uncovered background area.

In step S12, the mixture ratio calculator 104 calculates the mixtureratio α for each pixel included in the mixed area on the basis of theinput image and the area information. Details of the mixture ratiocalculating processing will be given later. The mixture ratio calculator104 supplies the calculated mixture ratio α to the foreground/backgroundseparator 105.

In step S13, the foreground/background separator 105 extracts theforeground components from the input image on the basis of the areainformation and the mixture ratio α to supply them to the motion bluradjusting unit 106 as the foreground component image.

In step S14, the motion blur adjusting unit 106 generates units ofprocessing which specify the positions of the image in pixels which arecontinuously arranged in the moving direction and which correspond toany of the uncovered background area, the foreground area and thecovered background area on the basis of the motion vector and the areainformation, and adjusts the amount of motion blur included in theforeground components corresponding to the units of processing. Theprocessing details of adjusting the amount of motion blur will beexplained later.

In step S15, the separation server 11 determines whether the processingis finished for the whole screen or not, and if it is determined thatthe processing is not finished for the whole screen, the procedurereturns to step S14, and the process of adjusting the amount of motionblur for the foreground components corresponding to the unit ofprocessing is repeated.

In step S15, if it is determined that the processing is finished for thewhole screen, the process ends.

Like this, separation server 11 can separate the foreground and thebackground to adjust the amount of motion blur included the foreground.That is, the separation server 11 can adjust the amount of motion blurincluded in the sample data that are the pixel values of pixels of theforeground.

Now, the respective configurations of the area specifying unit 103, themixture ratio calculator 104, the foreground/background separator 105and the motion blur adjusting unit 106 will be explained.

FIG. 45 is a block diagram showing an example of configuration of thearea specifying unit 103. The area specifying unit 103 of whichconfiguration is shown in FIG. 45 does not use the motion vector. Aframe memory 201 stores the input image in a frame unit. When the targetof processing is a frame #n, the frame memory 201 stores a frame #n−2two frames ahead of the frame #n, a frame #n−1 one frame ahead of theframe #n, a frame #n+1 one frame behind the frame #n and the frame #n+2two frames behind the frame #n.

A stationary/moving determining portion 202-1 reads out from the framememory 201 the pixel value of the pixel in the frame #n+2 located at thesame position as the position of the image in the pixel that is a targetof the area specification in the frame #n and the pixel value of thepixel in the frame #n+1 located at the same position as the position ofthe image in the pixel that is also a target of the area specificationin the frame #n, to calculate the absolute value of the differencebetween the read-out pixel values. The stationary/moving determiningportion 202-1 determines whether the absolute value of the differencebetween the pixel value in the frame #n+2 and the pixel value in theframe #n+1 is greater than the pre-set threshold value Th or not andsupplies the stationary/moving determination indicating motion status tothe area determining portion 203-1 if it is determined that the absolutevalue of the difference is greater than the threshold value Th. If it isdetermined that the absolute value of the difference between the pixelvalue in the frame #n+2 and the pixel value in the frame #n+1 is notgreater than the threshold value Th, the stationary/moving determiningportion 202-1 supplies the stationary/moving determination indicatingstationary status to the area determining portion 203-1.

A stationary/moving determining portion 202-2 reads out from the framememory 201 the pixel value of pixel in the frame #n+1 located at thesame position as the position of the image in the pixel that is a targetof the area specification in the frame #n and the pixel value of pixelthat is a target of the area specification in the frame #n, to calculatethe absolute value of the difference between the pixel values. Thestationary/moving determining portion 202-2 determines whether theabsolute value of the difference between the pixel value in the frame#n+1 and the pixel value in the frame #n is greater than the pre-setthreshold value Th or not and supplies the stationary/movingdetermination indicating motion status to the area determining portion203-1 and the area determining portion 203-2 if it is determined thatthe absolute value of the difference between the pixel values is greaterthan the threshold value Th. If it is determined that the absolute valueof the difference between the pixel value of the pixel in the frame #n+1and the pixel value of pixel in the frame #n is not greater than thethreshold value Th, the stationary/moving determining portion 202-1supplies the stationary/moving determination indicating stationarystatus to the area determining portion 203-1 and the area determiningportion 203-2.

The stationary/moving determining portion 202-3 reads out from the framememory 201 the pixel value of pixel that is a target of the areaspecification in the frame #n and the pixel value of pixel in the frame#n−1 located at the same position as the position in the image of thepixel that is a target of the area specification in the frame tocalculate the absolute value of the difference between the pixel values.The stationary/moving determining portion 202-3 determines whether theabsolute value of the difference between the pixel value in the frame #nand the pixel value in the frame #n−1 is greater than the pre-setthreshold value Th or not and supplies the stationary/movingdetermination indicating motion status to the area determining portion203-2 and the area determining portion 203-3 if it is determined thatthe absolute value of the difference between the pixel values is greaterthan the threshold value Th. If it is determined that the absolute valueof the difference between the pixel value of pixel in the frame #n andthe pixel value of pixel in the frame #n−1 is not greater than thethreshold value Th, the stationary/moving determining portion 202-3supplies the stationary/moving determination indicating stationarystatus to the area determining portion 203-2 and the area determiningportion 203-3.

The stationary/moving determining portion 202-4 reads out from the framememory 201 the pixel value of pixel in the frame #n−1 located at thesame position as the position in the image of the pixel that is a targetof the area specification in the frame #n and the pixel value of pixelin the frame #n−2 located at the same position as the position in theimage of the pixel that is a target of the area specification in theframe #n, to calculate the absolute value of the difference between theread-out pixel values. The stationary/moving determining portion 202-4determines whether the absolute value of the difference between thepixel value in the frame #n−1 and the pixel value in the frame #n−1 isgreater than the pre-set threshold value Th or not and supplies thestationary/moving determination indicating motion status to the areadetermining portion 203-3 if it is determined that the absolute value ofthe difference is greater than the threshold value Th. If it isdetermined that the absolute value of the difference between the pixelvalue of pixel in the frame #n−1 and the pixel value of pixel in theframe #n−2 is not greater than the threshold value Th, thestationary/moving determining portion 202-4 supplies thestationary/moving determination indicating stationary status to the areadetermining portion 203-3.

When the stationary/moving determination supplied from thestationary/moving determining portion 202-1 indicates stationary statusand the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates motion status, thearea determining portion 203-1 determines that the pixel being a targetof the area specification in the frame #n belongs to the uncoveredbackground area and sets “1” indicating that the pixel belongs to theuncovered background area to the uncovered background area determiningflag corresponding to the pixel of which area is determined.

When the stationary/moving determination supplied from thestationary/moving determining portion 202-1 indicates motion status orthe stationary/moving determination supplied from the stationary/movingdetermining portion 202-2 indicates stationary status, the areadetermining portion 203-1 determines that the pixel being a target ofthe area specification in the frame #n does not belong to the uncoveredbackground area and sets “0” indicating that the pixel does not belongto the uncovered background area to the uncovered background areadetermining flag corresponding to the pixel of which area is determined.

The area determining portion 203-1 supplies the uncovered backgroundarea determining flag to which “1” or “0” is set as described by theabove to the frame memory storing determining flag 204.

When the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates stationary statusand the stationary/moving determination supplied from thestationary/moving determining portion 202-3 indicates stationary status,the area determining portion 203-2 determines that the pixel being atarget of the area specification in the frame #n belongs to thestationary area and sets “1” indicating that the pixel belongs to thestationary area to the stationary area determining flag corresponding tothe pixel of which area is determined.

When the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates motion status orthe stationary/moving determination supplied from the stationary/movingdetermining portion 202-3 indicates motion status, the area determiningportion 203-2 determines that the pixel being a target of the areaspecification in the frame #n does not belong to the stationary area andsets “0” indicating that the pixel does not belong to the stationaryarea to the stationary area determining flag corresponding to the pixelof which area is determined.

The area determining portion 203-2 supplies the stationary areadetermining flag to which “1” or “0” is set as described by the above tothe frame memory storing determining flag 204.

When the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates motion status andthe stationary/moving determination supplied from the stationary/movingdetermining portion 202-3 indicates motion status, the area determiningportion 203-2 determines that the pixel being a target of the areaspecification in the frame #n belongs to the moving area and sets “1”indicating that the pixel belongs to the moving area to the moving areadetermining flag corresponding to the pixel of which area is determined.

When the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates stationary statusor the stationary/moving determination supplied from thestationary/moving determining portion 202-3 indicates stationary status,the area determining portion 203-2 determines that the pixel being atarget of the area specification in the frame #n does not belong to themoving area and sets “0” indicating that the pixel does not belong tothe moving area to the moving area determining flag corresponding to thepixel of which area is determined.

The area determining portion 203-2 supplies the moving area determiningflag to which “1” or “0” is set as described by the above to the framememory storing determining flag 204.

When the stationary/moving determination supplied from thestationary/moving determining portion 202-3 indicates motion status andthe stationary/moving determination supplied from the stationary/movingdetermining portion 202-4 indicates stationary status, the areadetermining portion 203-3 determines that the pixel being a target ofthe area specification in the frame #n belongs to the covered backgroundarea and sets “1” indicating that the pixel belongs to the coveredbackground area to the covered background area determining flagcorresponding to the pixel of which area is determined.

When the stationary/moving determination supplied from thestationary/moving determining portion 202-3 indicates stationary statusor the stationary/moving determination supplied from thestationary/moving determining portion 202-4 indicates motion status, thearea determining portion 203-2 determines that the pixel being a targetof the area specification in the frame #n does not belong to the coveredbackground area and sets “0” indicating that the pixel does not belongto the covered background area to the covered background areadetermining flag corresponding to the pixel of which area is determined.

The area determining portion 203-3 supplies the covered background areadetermining flag to which “1” or “0” is set as described by the above tothe frame memory storing determining flag 204.

The frame memory storing determining flag 204 stores the uncoveredbackground area determining flag supplied from the area determiningportion 203-1, the stationary area determining flag supplied from theare determining portion 203-2, the moving area determining flag suppliedfrom the area determining portion 203-2 and the covered background areadetermining flag supplied from the area determining portion 203-3,respectively.

The frame memory storing determining flag 204 supplies the uncoveredbackground area determining flag, the stationary area determining flag,the moving area determining flag and the covered background areadetermining flag stored therein to a synthesizer 205. The synthesizer205 generates the area information indicating that each pixel belongs toany of the uncovered background area, the stationary area, the movingarea and the covered background area on the basis of the uncoveredbackground area determining flag, the stationary area determining flag,the moving area determining flag and the covered background areadetermining flag supplied from the frame memory storing determining flag204 and then supplies it to the frame memory storing determining flag206.

The frame memory storing determining flag 206 stores the areainformation supplied from the synthesizer 205 and outputs the storedarea information.

Next, referring to FIGS. 46 to 50, an example of processing by the areaspecifying unit 103 will be explained.

When the object corresponding to the foreground is moving, the positionon the screen of the image corresponding to the object varies everyframe. As shown in FIG. 46, the image corresponding to the objectlocated at a position indicated by Yn(x, y) in the frame #n is locatedat Yn+1(x, y) in the frame #n+1.

FIG. 22 is a model diagram in which the pixel values of pixels beingadjacent in a row in the moving direction of the image corresponding tothe foreground object are expanded in the time direction. For example,when the moving direction of the image corresponding to the foregroundobject is horizontal on the screen, the model diagram in FIG. 47 shows amodel in which the pixel values of pixels being adjacent in one line aredeveloped in the time direction.

In FIG. 47, the lines in the frame #n are the same as those in the frame#n+1.

The foreground components corresponding to the object included in thesecond to the thirteenth pixels from the leftmost in the frame #n areincluded in the sixth to the seventeenth pixels from the leftmost in theframe #n+1.

In the frame #n, pixels belonging to the covered background area are theeleventh to thirteenth pixels from the leftmost and pixels belonging tothe uncovered background area are the second to fourth pixels from theleftmost. In the frame #n+1, pixels belonging to the covered backgroundarea are the fifteenth to seventeenth pixels from the leftmost andpixels belonging to the uncovered background area are the sixth toeighth pixels from the leftmost.

In the example shown in FIG. 47, since the foreground componentsincluded in the frame #n is moved by 4 pixels in the frame #n+1, theamount of movement v is defined as 4. The virtual division number is setat 4 corresponding to the amount of movement v.

Next, variation of the pixel values of pixels belonging to the mixedarea in front or back of the indicated frame will be explained.

In the frame #n that is shown in FIG. 48 and in which the background isstationary and the amount of movement v of the foreground is 4, pixelsbelonging to the covered background area are the fifteenth toseventeenth pixels from the leftmost. Since the amount of movement v is4, the fifteenth to seventeenth pixels in the frame #n−1 one frame aheadinclude only the background components and belong to the backgroundarea. Further, the fifteenth to seventeenth pixels in the frame #n−2further one frame ahead include only the background components andbelong to the background area.

Here, since the object corresponding to the background is stationary,the pixel value of the fifteenth pixel from the leftmost in the frame#n−1 is not changed from the pixel value of the fifteenth pixel from theleftmost in the frame #n−2. Similarly, the pixel value of the sixteenthpixel from the leftmost in the frame #n−1 is not changed from the pixelvalue of the sixteenth pixel from the leftmost in the frame #n−2 and thepixel value of the seventeenth pixel from the leftmost in the frame #n−1is not changed from the pixel value of the seventeenth pixel from theleftmost in the frame #n−2.

That is, since pixels in the frame #n−1 and the frame #n−2 correspondingto the pixels belonging to the covered background area in the frame #ncomprised only the background components and the pixel values thereofare not changed, the absolute value of the difference thereof is almost0. Therefore, the stationary/moving determination on the pixels in theframe #n−1 and the frame #n−2 corresponding to pixels belonging to themixed area in the frame #n is determined to be stationary by thestationary/moving determining portion 202-4.

Since pixels belonging to the covered background area in the frame #ninclude the foreground components, the pixel values thereof aredifferent from those in the case of including only the backgroundcomponents in the frame #n−1. Therefore, the stationary/movingdetermination on the pixels belonging to the mixed area in the frame #nand pixels in the corresponding frame #n−1 is determined to be moving bythe stationary/moving determining portion 202-3.

Like the above, when the stationary/moving determination resultindicating motion status is supplied from the stationary/movingdetermining portion 202-3 and the stationary/moving determination resultindicating stationary status is supplied from the stationary/movingdetermining portion 202-4, the area determining portion 203-3 determinesthat the corresponding pixel belongs to the covered background area.

In the frame #n that is shown in FIG. 49 and in which the background isstationary and the amount of movement v of the foreground is 4, thepixels belonging to the uncovered background area are the second tofourth pixels from the leftmost. Since the amount of movement v is 4,the second to fourth pixels in the frame #n+1 one frame behind includeonly the background components and belong to the background area.Further, the second to fourth pixels in the frame #n+2 further one framebehind include only the background components and belong to thebackground area.

Here, since the object corresponding to the background is stationary,the pixel value of the second pixel from the leftmost in the frame #n+2is not changed from the pixel value of the second pixel from theleftmost in the frame #n+1. Similarly, the pixel value of the thirdpixel from the leftmost in the frame #n+2 is not changed from the pixelvalue of the third pixel from the leftmost in the frame #n+1 and thepixel value of the fourth pixel from the leftmost in the frame #n+2 isnot changed from the pixel value of the fourth pixel from the leftmostin the frame #n+1.

That is, since pixels in the frame #n+1 and the frame #n+2 correspondingto the pixels belonging to the uncovered background area in the frame #ncomprised only the background components and the pixel values thereofare not changed, the absolute value of the difference thereof is almost0. Therefore, the stationary/moving determination on the pixels in theframe #n+1 and the frame #n+2 corresponding to pixels belonging to themixed area in the frame #n is determined to be stationary by thestationary/moving determining portion 202-1.

Since pixels belonging to the uncovered background area in the frame #ninclude the foreground components, the pixel values thereof aredifferent from those in the case of including only the backgroundcomponents in the frame #n+1. Therefore, the stationary/movingdetermination on the pixels belonging to the mixed area in the frame #nand pixels in the corresponding frame #n+1 is determined to be moving bythe stationary/moving determining portion 202-2.

Like the above, when the stationary/moving determination resultindicating motion status is supplied from the stationary/movingdetermining portion 202-2 and the stationary/moving determination resultindicating stationary status is supplied from the stationary/movingdetermining portion 202-1, the area determining portion 203-1 determinesthat the corresponding pixel belongs to the uncovered background area.

FIG. 50 shows determination conditions of the area specifying unit 103in the frame #n. When it is determined that a pixel in the frame #n−2located at the same position as the position in the image of a pixelthat is a target of determination in the frame #n and a pixel in theframe #n−1 located at the same position as the position in the image ofthe pixel that is a target of determination in the frame #n, arestationary and when it is determined that a pixel in the frame #n−1located at the same position as the position in the image of the pixelthat is a target of determination in the frame #n and the pixel in theframe #n are moving, the area specifying unit 103 determines that thepixel that is a target of determination in the frame #n belongs to thecovered background area.

When it is determined that a pixel in the frame #n−1 located at the sameposition as the position in the image of the pixel that is a target ofdetermination in the frame #n and the pixel in the frame #n, arestationary and when it is determined that the pixel in the frame #n andthe pixel in the frame #n+1 located at the same position as the positionin the image of the pixel that is a target of determination in the frame#n are stationary, the area specifying unit 103 determines that thepixel that is a target of determination in the frame #n belongs to thestationary area.

When it is determined that the pixel in the frame #n−1 located at thesame position as the position in the image of the pixel that is a targetof determination in the frame #n and the pixel in the frame #n, aremoving and when it is determined that the pixel in the frame #n and thepixel in the frame #n+1 located at the same position as the position inthe image of the pixel that is a target of determination in the frame #nare moving, the area specifying unit 103 determines that the pixel thatis a target of determination in the frame #n belongs to the moving area.

When it is determined that the pixel in the frame #n and the pixel inthe frame #n+1 located at the same position as the position in the imageof the pixel that is a target of determination in the frame #n, aremoving and when it is determined that the pixel in the frame #n+1located at the same position as the position in the image of the pixelthat is a target of determination in the frame #n and the pixel in theframe #n+2 located at the same position as the position in the image ofthe pixel that is a target of determination in the frame #n arestationary, the area specifying unit 103 determines that the pixel thatis a target of determination in the frame #n belongs to the uncoveredbackground area.

FIGS. 51A to 51D show examples of area specification result of the areaspecifying unit 103. In FIG. 51A, pixel determined to belong in thecovered background area is displayed in white color. In FIG. 51B, apixel determined to belong in the uncovered background area is displayedin white color.

In FIG. 51C, a pixel determined to belong in the moving area isdisplayed in white color. In FIG. 51D, a pixel determined to belong inthe stationary area is displayed in white color.

FIG. 52 shows as an image of the area information indicating the mixedarea from the area information output from the frame memory storingdeterminating flag 206. In FIG. 52, the pixel determined to belong inthe covered background area or the uncovered background area, that is,the pixel determined to belong to the mixed area is displayed in whitecolor. The area information indicating the mixed area output from theframe memory storing determinating flag 206 means a portion havingtexture surrounded by a portion not having texture in the mixed area andthe foreground area.

Next, referring to the flowchart of FIG. 53, the area specifyingprocessing of the area specifying unit 103 will be explained. In stepS201, the frame memory 201 obtains images of the frame #n−2 to the frame#n+2 including the frame #n that is a target of determination.

In step S202, the stationary/moving determining portion 202-3 determineswhether it is stationary or not for the pixel in the frame #n−1 and thepixel in the frame #n located at the same position and if it isdetermined to be stationary, the procedure proceeds to step S203. Thestationary/moving determining portion 202-2 determines whether it isstationary or not for the pixel in the frame #n and the pixel in theframe #n+1 located at the same position.

In step S203, when it is determined to be stationary for the pixel inthe frame #n and the pixel in the frame #n+1 located at the sameposition, the procedure proceeds to step S204 and the area determiningportion 203-2 sets “1” indicating the pixel belongs to the stationaryarea to the stationary area determinating flag corresponding to thepixel of which area is determined. The area determining portion 203-2supplies the stationary area determinating flag to the frame memorystoring determinating flag 204 and the procedure proceeds to step S205.

When it is determined to be moving for the pixel in the frame #n−1 andthe pixel in the frame #n position located at the same position in stepS202 or when it is determined to be moving for the pixel in the frame #nand the pixel in the frame #n+1 position located at the same position instep S203, since the pixel in the frame #n does not belong to thestationary area, the processing in step S204 is skipped and theprocedure proceeds to step S205.

In step S205, the stationary/moving determining portion 202-3 determineswhether it is moving or not for the pixel in the frame #n−1 and thepixel in the frame #n located at the same position and if it isdetermined to be moving, the procedure proceeds to step S206. Thestationary/moving determining portion 202-2 determines whether it ismoving or not for the pixel in the frame #n and the pixel in the frame#n+1 located at the same position.

In step S206, when it is determined to be moving for the pixel in theframe #n and the pixel in the frame #n+1 located at the same position,the procedure proceeds to step S207 and the area determining portion203-2 sets “1” indicating the pixel belongs to the moving area to themoving area determinating flag corresponding to the pixel of which areais determined. The area determining portion 203-2 supplies the movingarea determinating flag to the frame memory storing determinating flag204 and the procedure proceeds to step S208.

When it is determined to be stationary for the pixel in the frame #n−1and the pixel in the frame #n position located at the same position instep S205 or when it is determined to be stationary for the pixel in theframe #n and the pixel in the frame #n+1 position located at the sameposition in step S206, since the pixel in the frame #n does not belongto the moving area, the processing in step S207 is skipped and theprocedure proceeds to step S208.

In step S208, the stationary/moving determining portion 202-4 determineswhether it is stationary or not for the pixel in the frame #n−2 and thepixel in the frame #n−1 located at the same position and if it isdetermined to be stationary, the procedure proceeds to step S209. Thestationary/moving determining portion 202-3 determines whether it ismoving or not for the pixel in the frame #n−1 and the pixel in the frame#n located at the same position.

In step S209, when it is determined to be moving for the pixel in theframe #n−1 and the pixel in the frame #n located at the same position,the procedure proceeds to step S210 and the area determining portion203-3 sets “1” indicating the pixel belongs to the covered backgroundarea to the covered background area determinating flag corresponding tothe pixel of which area is determined. The area determining portion203-3 supplies the covered background area determinating flag to theframe memory storing determinating flag 204 and the procedure proceedsto step S211.

When it is determined to be moving for the pixel in the frame #n−2 andthe pixel in the frame #n−1 position located at the same position instep S208 or when it is determined to be stationary for the pixel in theframe #n−1 and the pixel in the frame #n position located at the sameposition in step S209, since the pixel in the frame #n does not belongto the covered background area, the processing in step S210 is skippedand the procedure proceeds to step S211.

In step S211, the stationary/moving determining portion 202-2 determineswhether it is moving or not for the pixel in the frame #n and the pixelin the frame #n+1 located at the same position and if it is determinedto be moving, the procedure proceeds to step S212. The stationary/movingdetermining portion 202-1 determines whether it is stationary or not forthe pixel in the frame #n+1 and the pixel in the frame #n+2 located atthe same position.

In step S212, when it is determined to be stationary for the pixel inthe frame #n+1 and the pixel in the frame #n+2 located at the sameposition, the procedure proceeds to step S213 and the area determiningportion 203-1 sets “1” indicating the pixel belongs to the uncoveredbackground area to the uncovered background area determinating flagcorresponding to the pixel of which area is determined. The areadetermining portion 203-1 supplies the uncovered background areadeterminating flag to the frame memory storing determinating flag 204and the procedure proceeds to step S214.

When it is determined to be stationary for the pixel in the frame #n andthe pixel in the frame #n+1 position located at the same position instep S211 or when it is determined to be moving for the pixel in theframe #n+1 and the pixel in the frame #n+2 position located at the sameposition in step S212, since the pixel in the frame #n does not belongto the uncovered background area, the processing in step S213 is skippedand the procedure proceeds to step S214.

In step S214, the area specifying unit 103 determines whether areaspecification has been performed for all the pixels in the frame #n ornot and if it is determined that the area specification has not beenperformed for all the pixels in the frame #n, the procedure returns tostep S202 and the area specifying processing is repeated for otherpixels.

In step S214, if it is determined that the area specification has beenperformed for all the pixels in the frame #n, the procedure proceeds tostep S215 and the synthesizer 205 generates the area informationindicating the mixed area on the basis of the uncovered background areadeterminating flag and the covered background area determinating flagstored in the frame memory storing determinating flag 204, generates thearea information indicating that each pixel belongs to which of theuncovered background area, the stationary area, the moving area and thecovered background area and sets the generated area information in theframe memory storing determinating flag 206. Then, the processing ends.

Like the above, the area specifying unit 103 can generate the areainformation indicating that the each pixel included in the frame belongsto the moving area, the stationary area, the uncovered background areaor the covered background area.

Further, the area specifying unit 103 may generate the area informationcorresponding to the mixed area by applying logical sum to the areainformation corresponding to the uncovered background area and coveredbackground area and may generate the area information comprised of flagsindicating that the each pixel included in the frame belongs to themoving area, the stationary area or the mixed area.

When the object corresponding to the foreground has texture, the areaspecifying unit 103 can specify the moving area more accurately.

The area specifying unit 103 can output the area information indicatingthe moving area as the area information indicating the foreground areaand the area information indicating the stationary area as the areainformation indicating the background area.

Further, although it has been explained that the object corresponding tothe background is stationary, even when the image corresponding to thebackground area includes motion, the area specifying processingdescribed above can be applied. For example, when an image correspondingto the background area is moving constantly, the area specifying unit103 shifts the whole image corresponding to this movement and processesit similarly to the case that the object corresponding to the backgroundis stationary. Further, when an image corresponding to the backgroundarea includes other movements locally, the area specifying unit 103selects pixels corresponding to the movements and performs theaforementioned processing.

FIG. 54 is a block diagram illustrating another example of configurationof the area specifying unit 103. The area specifying unit 103 shown inFIG. 54 does not use the motion vector. A background image generator 301generates the background image corresponding to the input image andsupplies the generated background image to a binary object imageextracting portion 302. The background image generator 301 extracts, forexample, an image object corresponding to the object of the backgroundincluded in the input image to generate the background image.

An example of a model in which the pixel values of pixels being adjacentin a row in the moving direction of an image corresponding to theforeground object are expanded in the time direction is shown in FIG.55. For example, when the moving direction of the image corresponding tothe foreground object is horizontal on the screen, the model diagramshown in FIG. 55 shows a model in which the pixel values of pixels beingadjacent in one line are expanded in the time direction.

In FIG. 55, a line in a frame #n is the same as that in a frame #n−1 anda frame #n+1.

In the frame #n, the foreground components corresponding to the objectincluded in the sixth to seventeenth pixels from the leftmost areincluded in the second to thirteenth pixels from the leftmost in theframe #n−1 and in the tenth to twenty first pixels from the leftmost inthe frame #n+1.

In the frame #n−1, the pixels belonging to the covered background areaare the eleventh to thirteenth pixels from the leftmost and the pixelsbelonging to the uncovered background area are the second to fourthpixels from the leftmost. In the frame #n, the pixels belonging to thecovered background area are the fifteenth to seventeenth pixels from theleftmost pixels belonging to the uncovered background area are the sixthpixels to eighth pixels. In the frame #n+1, the pixels belonging to thecovered background area are the nineteenth to twenty first pixels fromthe leftmost and the pixels belonging to the uncovered background areaare the tenth to twelfth pixels from the leftmost.

In the frame #n−1, the pixels belonging to the background area are thefirst pixel and the fourteenth to twenty first pixels from the leftmost.In the frame #n, the pixels belonging to the background area are thefirst to fifth pixels and the eighteenth to twenty first pixels from theleftmost. In frame #n+1, the pixels belonging to the background area arethe first to ninth pixels from the leftmost.

An example of the background image corresponding to the example of FIG.55 which is generated from background generator 301 is shown in. FIG.56. The background image is comprised of pixels corresponding to thebackground object and does not include the image componentscorresponding to the foreground object.

The binary object image extracting portion 302 generates the binaryobject image on the basis of the correlation between the backgroundimage and the input image and supplies the generated binary object imageto a time change detector 303.

FIG. 57 is a block diagram showing configuration of the binary objectimage extracting portion 302. A correlation value calculator 321calculates the correlation between the input image and the backgroundimage supplied from the background image generator 301 and generates andsupply the correlation value to a threshold value processor 322.

The correlation value calculator 321 applies equation (4) to, forexample, 3×3 blocks in the background image where X4 is centered asshown in FIG. 58A and 3×3 blocks in the input image, corresponding tothe blocks in the background image, where Y4 is centered as shown inFIG. 58B to calculate the correlation value corresponding to Y4.

$\begin{matrix}{{{CORRELATION}\mspace{14mu} {VALUE}} = \frac{\sum\limits_{i = 0}^{8}\; {\left( {X_{i} - \overset{\_}{X}} \right){\sum\limits_{i = 0}^{8}\; \left( {Y_{i} - \overset{\_}{Y}} \right)}}}{\sqrt{\begin{matrix}{\sum\limits_{i = 0}^{8}\; {\left( {X_{i} - \overset{\_}{X}} \right)^{2} \cdot}} \\{\sum\limits_{i = 0}^{8}\; \left( {Y_{i} - \overset{\_}{Y}} \right)^{2}}\end{matrix}}}} & (4) \\{\overset{\_}{X} = \frac{\sum\limits_{i = 0}^{8}\; X_{i}}{9}} & (5) \\{\overset{\_}{Y} = \frac{\sum\limits_{i = 0}^{8}\; Y_{i}}{9}} & (6)\end{matrix}$

The correlation value calculator 321 supplies the correlation valuecalculated for the respective pixels as described by the above to thethreshold vale processor 322. Further, the correlation value calculator321 may apply equation (7) to, for example, 3×3 blocks in the backgroundimage where X4 is centered as shown in FIG. 59A and 3×3 blocks in theinput image, corresponding to the blocks in the background image, whereY4 is centered as shown in FIG. 59B to calculate the absolute value ofdifference corresponding to Y4.

$\begin{matrix}{{{SUM}\mspace{14mu} {OF}\mspace{14mu} {ABSOLUTE}\mspace{14mu} {VALUE}\mspace{14mu} {OF}\mspace{14mu} {DIFFERENCE}} = {\sum\limits_{i = 0}^{8}\; {\left( {X_{i} - Y_{i}} \right)}}} & (7)\end{matrix}$

The correlation value calculator 321 supplies the absolute value of thedifference calculated by the process described above as the correlationvalue to the threshold value processor 322.

The threshold value processor 322 compares the pixel value of thecorrelating image to the threshold value th0, sets 1 to the pixel valueof the binary object image when the correlation value is not greaterthan the threshold value th0, sets 0 to the pixel value of the binaryobject image when the correlation value is greater than the thresholdvalue th0 and outputs the binary object image to which pixel value 0 or1 is set. The threshold value processor 322 may previously store thethreshold value th0 or may use the externally input threshold value th0.

FIG. 60 shows an example of a binary object image corresponding to themodel of the input image shown in FIG. 55. In the binary object image, 0is set to the pixel value of pixel that has a high correlation to thebackground image.

FIG. 61 is a block diagram showing a configuration of the time changedetector 303. When performing the area determination for the pixels inthe frame #n, the frame memory 341 stores the binary object image of theframe #n−1, the frame #n and the frame #n+1 supplied from the binaryobject image extracting portion 302.

The area determining portion 342 determines area for each pixel in theframe #n on the basis of the binary object image of the frame #n−1, theframe #n and the frame #n+1 stored in the frame memory 341 to generateand output the area information.

FIG. 62 is a diagram illustrating a determination process of the areadetermining portion 342. When the indicated pixel of the binary objectimage in the frame #n is 0, the area determining portion 342 determinesthat the indicated pixel in the frame #n belongs to the background area.

When the indicated pixel of the binary object image in the frame #n is1, the indicated pixel of the binary object image in the frame #n−1 is 1and the indicated pixel of the binary object image in the frame #n+1 is1, the area determining portion 342 determines that the indicated pixelin the frame #n belongs to the foreground area.

When the indicated pixel of the binary object image in the frame #n is 1and the indicated pixel of the binary object image in the frame #n−1 is0, the area determining portion 342 determines that the indicated pixelin the frame #n belongs to the covered background area.

When the indicated pixel of the binary object image in the frame #n is 1and the indicated pixel of the binary object image in the frame #n+1 is0, the area determining portion 342 determines that the indicated pixelin the frame #n belongs to the uncovered background area.

FIG. 63 shows an example of the time change detector 303 performingdetermination process with respect to the binary object imagecorresponding to the model of the input image shown in FIG. 55. Sincethe pixel in the frame #n corresponding to the binary object image is 0,the time change detector 303 determines that the first to fifth pixelsfrom the leftmost in the frame #n belong to the background area.

Since the pixel in the frame #n of the binary object image is 1 and thecorresponding pixel in the frame #n+1 is 0, the time change detector 303determines that the sixth to ninth pixels from the leftmost belong tothe uncovered background area.

Since the pixel in the frame #n of the binary object image is 1, thecorresponding pixel in the frame #n−1 is 1 and the corresponding pixelin the frame #n+1 is 1, the time change detector 303 determines that thetenth to thirteenth pixels from the leftmost belong to the foregroundarea.

Since the pixel in the frame #n of the binary object image is 1 and thecorresponding pixel in the frame #n−1 is 0, the time change detector 303determines that the fourteenth to seventeenth pixels from the leftmostbelong to the covered background area.

Since the corresponding pixel in the frame #n of the binary object imageis 0, the time change detector 303 determines that the eighteenth totwenty first pixels from the leftmost belong to the background area.

Next, referring to a flowchart of FIG. 64, the area specifyingprocessing of the area determining unit 103 will be explained. In stepS301, the background image generator 301 of the area determining unit103 extracts, for example, the image object corresponding to thebackground object included in the input image on the basis of the inputimage to generate the background image and supplies the generatedbackground image to the binary object image extracting portion 302.

In step S302, the binary object image extracting portion 302 calculatesthe correlation value between the input image and the background imagesupplied from the background image generator 301, for example, by usingcalculations which were explained by referring to FIG. 58A and FIG. 58B.In step S303, the binary object image extracting portion 302 calculatesthe binary object image from the correlation value and the thresholdvalue th0, for example, by comparing the correlation value with thethreshold value th0.

In step S304, the time change detector 303 performs the area determiningprocessing to complete the processing.

Referring to a flowchart of FIG. 65, the details of the area determiningprocessing corresponding to step S304 will be explained. In step S321,the area determining portion 342 of the time change detector 303determines whether the indicated pixel in the frame #n stored in theframe memory 341 is 0 or not and if it is determined that the indicatedpixel in the frame #n is 0, the procedure proceeds to step S322. Then,the area determining portion 342 determines that the noticed pixel inthe frame #n belongs to the background area to complete the processing.

In step S321, if it is determined that the indicated pixel in the frame#n is 1, the procedure proceeds to step S323. The area determiningportion 342 of the time change detector 303 determines whether theindicated pixel in the frame #n stored in the frame memory 341 is 1 andthe corresponding pixel in the frame #n−1 is 0 or not, and if it isdetermined that the indicated pixel in the frame #n is 1 and thecorresponding pixel in the frame #n−1 is 0, the procedure proceeds tostep S324. Then, the area determining portion 342 determines that theindicated pixel in the frame #n belongs to the covered background areato finish the processing.

In step S323, if it is determined that the indicated pixel in the frame#n is 0 or the corresponding pixel in the frame #n−1 is 1, the procedureproceeds to step S325. The area determining portion 342 of the timechange detector 303 determines whether the indicated pixel in the frame#n stored in the frame memory 341 is 1 and the corresponding pixel inthe frame #n+1 is 0 or not, and if it is determined that the noticedpixel in the frame #n is 1 and the corresponding pixel in the frame #n+1is 0, the procedure proceeds to step S326. Then, the area determiningportion 342 determines that the indicated pixel in the frame #n belongsto the uncovered background area and finishes the processing.

In step S325, if it is determined that the indicated pixel in the frame#n is 0 or the corresponding pixel in the frame #n+1 is 1, the procedureproceeds to step S327. The area determining portion 342 of the timechange detector 303 determines that the noticed pixel in the frame #nbelongs to the foreground area and finishes the processing.

Like the above, the area specifying unit 103 can specify which of theforeground area, the background area, the covered background area andthe uncovered background area that the pixels of the input image belongon the basis of the correlation value between the input image and thecorresponding background image, and generates the area informationcorresponding to the specification result.

FIG. 66 is a block diagram showing other configuration of the areaspecifying unit 103. The area specifying unit 103 shown in FIG. 66 usesthe motion vector supplied from the motion detector 102 and thepositional information thereof. Portions equal to those in FIG. 54 areindicated by the same reference numerals and explanation thereof will beomitted.

A robust processing portion 361 generates a binary object image of whichrobust processing is performed on the basis of the binary object imageof N frames supplied from the binary object image extracting portion 302and outputs it to the time change detector 303.

FIG. 67 is a block diagram illustrating a configuration of the robustprocessing portion 361. A motion compensator 381 compensates for motionof the binary object image of N frames on the basis of the motion vectorsupplied from the motion detector 102 and the positional informationthereof, and outputs the binary object image of which motion iscompensated to a switch 382.

Referring to examples of FIG. 68 and FIG. 69, a motion compensationprocess of the motion compensator 381 will be explained. For example,when an area in a frame #n is determined, if the binary object image ofa frame #n−1, a frame #n and a frame #n+1 exemplified in FIG. 68 isinput, the motion compensator 381 compensates the binary object image ofthe frame #n−1 and the binary object image of the frame #n+1, asexemplified in FIG. 69, on the basis of the motion vector supplied fromthe motion detector 102 and supplies the binary object image of whichmotion is compensated to the switch 382.

The switch 382 outputs the binary object image of the first frame inwhich motion is compensated to the frame memory 383-1 and outputs thebinary object image, of the second frame in which motion is compensatedto the frame memory 383-2. Similarly, the switch 382 outputs each of thebinary object images of the third to N-1-th frames in which motion iscompensated to one of the frame memory 383-3 to 383-(N-1) and outputsthe binary object image of the Nth frame in which motion is compensatedto the frame memory 383-N.

The frame memory 383-1 stores the binary object image of the first framein which motion is compensated and outputs the stored binary objectimage to the weighting portion 384-1. The frame memory 383-2 stores thebinary object image of the second frame in which motion is compensatedand outputs the stored binary object image to the weighting portion384-2.

Similarly, each of the frame memory 383-3 to 383-(N-1) stores one of thebinary object images of the third to (N-1)th frames of which motion iscompensated and outputs the stored binary object image to one ofweighting portions 384-3 to 384-(N-1). The frame memory 383-N stores thebinary object image of Nth frame of which motion is compensated andoutputs the stored binary object image to the weighting portion 384-N.

The weighting portion 384-1 multiplies the binary object image of thefirst frame supplied from the frame memory 383-1 of which motion iscompensated, by predetermined weight w1 to supply it to the accumulator385. The weighting portion 384-2 multiplies the pixel value of thebinary object image of the second frame supplied from the frame memory383-2 of which motion is compensated by a predetermined weight w2 tosupply it to the accumulator 385.

Similarly, each of the weighting portion 384-3 to the weighting portion384-(N-1) multiplies the pixel value of the binary object image of oneof the third to (N-1)th frames supplied from one of the frame memories383-3 to 383-(N-1) of which motion is compensated, by one of weight w3to weight w(N-1) to supply it to the accumulator 385. The weightingportion 384-N multiplies the pixel value of the binary object image ofthe Nth frame supplied from the frame memory 383-N of which motion iscompensated, by a predetermined weight wN to supply it to theaccumulator 385.

The accumulator 385 accumulates the pixel values corresponding to thebinary object images of the first to Nth frames respectively multipliedby one of the weights w1 to wN of which motions are compensated andgenerates the binary object image by comparing the accumulated pixelvalues with the predetermined threshold value th0.

Like the above, since the robust processing portion 361 generates thebinary object images of which robust processing is performed from Nbinary object images and supplies it to the time change detector 303,even when noises are included in the input image, the area specifyingunit 103 of which the configuration is shown in FIG. 66 can perform thearea specification more accurately than the case shown in FIG. 54.

Next, referring to a flowchart of FIG. 70, the area specifying processof the area specifying unit 103 of which the configuration is shown inFIG. 66 will be explained. The steps of step S341 to step S343 are equalto those of step S301 to step S303 and thus, explanation thereof will beomitted.

In step S344, the robust processing portion 361 performs the robustprocessing.

In step S345, the time change detector 303 performs the area specifyingprocess and finishes the procedure. Since the detailed process in stepS345 is similar to the process explained by referring to the flowchartof FIG. 65, explanation thereof will be omitted.

Next, referring to a flowchart of FIG. 71, the details of the robustprocessing corresponding to the processing of step S344 in FIG. 70 willbe explained. In step S361, a motion compensator 381 performs the motioncompensating process for the input binary object image on the basis ofthe motion vector supplied from the motion detector 102 and thepositional information thereof. In step S362, one of the frame memories383-1 to 383-N stores the motion compensated binary object imagesupplied through the switch 382.

In step S363, the robust processing portion 361 determines whether Nbinary object images are stored or not and if it is determined that Nbinary object images are not stored, the procedure returns to step S361and the motion compensating process for the binary object image and thestoring process for the binary object image are repeated. In step S363,if it is determined that N binary object images are stored, theprocedure proceeds to step S364. Each of the weighting portions 384-1 to384-N multiplies each of the N binary object images by one of theweights w1 to wN for weighting.

In step S365, an accumulator 385 accumulates the weighted N binaryobject images.

In step S366, the accumulator 385 generates a binary object image fromthe accumulated image, for example, by comparing to the predeterminedthreshold value th1 and finishes the procedure.

Like the above, the area specifying unit 103 of which the configurationis shown in FIG. 66 can generate the area information on the basis ofthe binary object image that is robust processed.

As described above, the area specifying unit 103 can generate the areainformation indicating that each pixel included in the frame belongs tothe moving area, the stationary area, the uncovered background area orthe covered background area.

FIG. 72 is a block diagram illustrating an example of a configuration ofthe mixture ratio calculator 104. An estimated mixture ratio processor401 calculates the mixture ratio of each pixel by using the calculationcorresponding to a model of the covered background area on the basis ofthe input image and supplies the calculated estimated mixture ratio tothe mixture ratio determining portion 403.

An estimated mixture ratio processor 402 calculates the mixture ratio ofeach pixel by using the calculation corresponding to the model of theuncovered background area on the basis of the input image and suppliesthe calculated estimated mixture ratio to the mixture ratio determiningportion 403.

Since it is supposed that the object corresponding to the foreground ismoving at a uniform speed for the shutter time, the mixture ratio α ofpixel belonging to the mixed area has the following feature. That is,the mixture ratio α is linearly varied corresponding to the positionalvariation of a pixel. If the positional variation of a pixel isone-dimensional, the variation in mixture ratio α can be expressed by alinear line and if the positional variation of pixel is two-dimensional,the variation in mixture ratio α can be expressed by a plane.

Further, since time interval for one frame is short, it can be supposedthat the object corresponding to the foreground is a rigid body and ismoving at a uniform speed.

In this case, the slope of mixture ratio α is reciprocal ratio to theamount of movement v in the shutter time of the foreground.

An example of an ideal mixture ratio α is shown in FIG. 73. The slope Iof the ideal mixture ratio α in the mixed area can be expressed as areciprocal number of the amount of movement v.

As shown in FIG. 73, the ideal mixture ratio α is 1 in the backgroundarea, is 0 in the foreground area and is greater than 0 and less than 1in the mixed area.

In the example of FIG. 74, the pixel value C06 of the seventh pixel fromthe leftmost in the frame #n can be expressed as equation (8) by usingthe pixel value P06 of the seventh pixel in the frame #n−1.

$\begin{matrix}\begin{matrix}{{C\; 06} = {{B\; {06/v}} + {B\; {06/v}} + {F\; {01/v}} + {F\; {02/v}}}} \\{= {{P\; {06/v}} + {P\; {06/v}} + {F\; {01/v}} + {F\; {02/v}}}} \\{= {{{{2/v} \cdot P}\; 06} + {\sum\limits_{i = 1}^{2}{F_{i}/v}}}}\end{matrix} & (8)\end{matrix}$

In equation (8), the pixel value C06 is expressed as the pixel value Mof pixel in the mixed area and the pixel value P06 is expressed as thepixel value B of pixel in the background area. That is, the pixel valueM of pixel in the mixed area and the pixel value B of pixel in thebackground area can be expressed by equation (9) and equation (10),respectively.

M=C06   (9)

B=P06   (10)

2/v in equation (8) corresponds to the mixture ratio α. Since the amountof movement v is defined as 4, the mixture ratio α of the seventh pixelfrom the leftmost in the frame #n is 0.5.

As described above, by considering the pixel value C in the indicatedframe #n as the pixel value in the mixed area and the pixel value P inthe frame #n−1 ahead of the frame #n as the pixel value in thebackground area, the equation (3) expressing the mixture ratio α can beexpressed by equation (11).

C=α·P+f   (11)

f in equation (11) is sum Σ_(i)Fi/v of the foreground componentsincluded in the indicated pixel. Variables included in equation (11) aretwo of the mixture ratio α and the sum f of the foreground components.

Similarly, in the uncovered background area, the amount of movement v is4 and the virtual division number in the time direction is 4. A model inwhich the pixel values are expanded in the time direction is shown inFIG. 75.

In the uncovered background area, similar to the expressions of theaforementioned covered background area, by considering the pixel value Cof pixel in the indicated frame #n as the pixel value in the mixed areaand the pixel value N in the frame #n+1 behind the frame #n as the pixelvalue in the background area, the equation (3) expressing the mixtureratio α can be expressed by equation (12).

C=α·N+f   (12)

Further, although it has been explained by considering that thebackground object is stationary, even when the background object ismoving, equation (8) to equation (12) can be applied by using the pixelvalue of pixel at a position corresponding to the amount of movement vof the background. For example, in FIG. 74, when the amount of movementv of the object corresponding to the background is 2, the virtualdivision number is 2 and when the object corresponding to the backgroundis moving toward the right side in the drawing, the pixel value B ofpixel in the background area in equation (10) becomes the pixel valueP04.

Since each of equation (11) and equation (12) includes two variables,the mixture ratio α cannot be obtained by these equations. Here, sincethe image has a strong correlation in space, the adjacent pixels arealmost equal.

Therefore, because the foreground components have a strong correlationin space, by changing the equations to derive the sum f of theforeground components from the front or next frame, the mixture ratio αis obtained.

The pixel value Mc of the seventh pixel from the leftmost in the frame#n in FIG. 76 can be expressed by equation (13).

$\begin{matrix}{M_{c} = {{{\frac{2}{v} \cdot B}\; 06} + {\sum\limits_{i = 11}^{12}{F_{i}/v}}}} & (13)\end{matrix}$

2/v of the first term of right side of equation (13) corresponds to themixture ratio α. The second term of right side of equation (13) can beexpressed by equation (14) using the pixel value in the next frame #n+1.

$\begin{matrix}{{\sum\limits_{i = 11}^{12}\; {F_{i}/v}} = {\beta \cdot {\sum\limits_{i = 7}^{10}\; {F_{i}/v}}}} & (14)\end{matrix}$

Here, using the spatial correlation of the foreground components,equation (15) can be obtained.

F=F05=F06=F07=F08=F09=F10=F11=F12   (15)

Equation (14) can be replaced with equation (16) by using equation (15).

$\begin{matrix}\begin{matrix}{{\sum\limits_{i = 11}^{12}\; {F_{i}/v}} = {\frac{2}{v} \cdot F}} \\{{= \beta}{\cdot \frac{4}{v} \cdot F}}\end{matrix} & (16)\end{matrix}$

Consequently, β can be expressed by equation (17).

β=2/4   (17)

In general, if it is supposed that the foreground componentscorresponding to the mixed area as shown in equation (15) are equal,equation (18) can be established for all the pixels in the mixed area,in view of relation of the interior division ratio.

β=1−α  (18)

If the equation (18) is established, equation (11) can be obtained asshown in equation (19).

$\begin{matrix}\begin{matrix}{{C = \alpha}{{\cdot P} + f}} \\{= {{\alpha \cdot P} + {\left( {1 - \alpha} \right) \cdot {\sum\limits_{i = \gamma}^{\gamma + V - 1}\; {F_{i}/v}}}}} \\{= {{\alpha \cdot P} + {\left( {1 - \alpha} \right) \cdot N}}}\end{matrix} & (19)\end{matrix}$

Similarly, if the equation (18) is established, equation (12) can beobtained as shown in equation (20).

$\begin{matrix}\begin{matrix}{{C = \alpha}{{\cdot N} + f}} \\{= {{\alpha \cdot N} + {\left( {1 - \alpha} \right) \cdot {\sum\limits_{i = \gamma}^{\gamma + V - 1}\; {F_{i}/v}}}}} \\{= {{\alpha \cdot N} + {\left( {1 - \alpha} \right) \cdot P}}}\end{matrix} & (20)\end{matrix}$

In equation (19) and equation (20), since C, N and P are the known pixelvalues, the only variable in equation (19) and equation (20) is themixture ratio α. The relationship of C, N and P in equation (19) andequation (20) is shown in FIG. 77. C is the pixel value of the indicatedpixel in the frame #n which calculates the mixed ratio α. N is the pixelvalue of pixel in the frame #n+1 of which the spatial positioncorresponds to that of the indicated pixel. P is the pixel value ofpixel in the frame #n−1 of which the spatial position corresponds tothat of the indicated pixel.

Therefore, since one variable is included respectively in equation (19)and equation (20), the mixture ratio α can be calculated using the pixelvalues of pixels in three frames. Conditions for calculating appropriatemixture ratio by solving equation (19) and equation (20) is that thepixel values of pixels are constant, the foreground componentscorresponding to the mixed area are equal, that is, the pixels beinglocated on the interface of the image object corresponding to the movingdirection of the foreground object in the foreground image object pickedup when the foreground object is stationary, a twice number of pixelsfor the amount of movement v are adjacent in a row.

As described above, the mixture ratio α of pixels belonging to thecovered background area is calculated by equation (21) and the mixtureratio α of pixels belonging to the uncovered background area iscalculated by equation (22).

α=(C−N)/(P−N)   (21)

α=(C−P)/(N−P)   (22)

FIG. 78 is a block diagram showing a configuration of the estimatedmixture ratio processor 401. The frame memory 421 stores the input imagein frame unit and supplies a frame one behind the frame input as theinput image, to the frame memory 422 and the mixture ratio calculator423.

The frame memory 422 stores the input image in frame unit and suppliesthe frame one behind the frame supplied from the frame memory 421, tothe mixture ratio calculator 423.

Therefore, when the frame #n+1 as the input image is input to themixture ratio calculator 423, the frame memory 421 supplies the frame #nto the mixture ratio calculator 423 and the frame memory 422 suppliesthe frame #n−1 to the mixture ratio calculator 423.

The mixture ratio calculator 423 calculates the estimated mixture ratioof the noticed pixels by using calculation shown in equation (21) on thebasis of the pixel value C of the indicated pixel in the frame #n, thepixel value N of pixel in the frame #n+1 of which the spatial positioncorresponds to that of the indicated pixel, and on the basis of thepixel value P of pixel in the frame #n−1 of which the spatial positioncorresponds to that of the indicated pixel, and outputs the calculatedestimated mixture ratio. For example, when the background is stationary,the mixture ratio calculator 423 calculates the estimated mixture ratioof the indicated pixel on the basis of the pixel value C of theindicated pixel in the frame #n, the pixel value N of pixel in the frame#n+1 of which the position is equal to that of the indicated pixel, andthe pixel value P of pixel in the frame #n−1 of which the position isequal to that of the indicated pixel, and outputs the calculatedestimated mixture ratio.

Like the above, the estimated mixture ratio processor 401 can calculatethe estimated mixture ratio on the basis of the input image and supplyit to the mixture ratio determining portion 403.

Further, since the estimated mixture ratio processor 402 has the samefunction as the estimated mixture ratio processor 401, except that theestimated mixture ratio processor 402 calculates the estimated mixtureratio of the indicated pixel by using calculation expressed by equation(22), on the contrary to the estimated mixture ratio processor 401 whichcalculates the estimated mixture ratio of the indicated pixel by usingcalculation expressed by equation (21), explanation thereof will beomitted.

FIG. 79 is a diagram illustrating an example of an estimated mixtureratio calculated by the estimated mixture ratio processor 401. Theestimated mixture ratio shown in FIG. 79 indicates the result when theamount of movement v of the foreground corresponding to the objectmoving at uniform speed is 11, with respect to a line.

It can be known that the estimated mixture ratio is almost linearlyvaried, as shown in the mixed area of FIG. 73.

Returning to FIG. 72, the mixture ratio determining portion 403 sets themixture ratio α on the basis of the area information supplied from thearea specifying unit 103, indicating which of the foreground area, thebackground area, the covered background area and the uncoveredbackground area, the pixel that is a target of calculation of themixture ratio α belongs to. The mixture ratio determining portion 403sets 0 to the mixture ratio α when the target pixel belongs to theforeground area, sets 1 to the mixture ratio a when the target pixelbelongs to the background area, sets the estimated mixture ratiosupplied from the estimated mixture ratio processor 401 to the mixtureratio a when the target pixel belongs to the covered background area,and sets the estimated mixture ratio supplied from the estimated mixtureratio processor 402, to the mixture ratio a when the target pixelbelongs to the uncovered background area. The mixture ratio determiningportion 403 outputs the mixture ratio a set on the basis of the areainformation.

FIG. 80 is a block diagram illustrating another configuration of themixture ratio calculator 104. The selector 441 supplies the pixelsbelonging to the covered background area and the pixels in the front andnext frames corresponding to the pixels to the estimated mixture ratioprocessing portion 442 on the basis of the area information suppliedfrom the area specifying unit 103. The selector 441 supplies the pixelsbelonging to the uncovered background area and the pixels in the frontand next frames corresponding to the pixels to the estimated mixtureratio processor 443 on the basis of the area information supplied fromthe area specifying unit 103.

The estimated mixture ratio processor 442 calculates the estimatedmixture ratio of the indicated pixel belonging to the covered backgroundarea by using calculation by equation (21) on the basis of the pixelvalue input from the selector 441, and supplies the calculated estimatedmixture ratio to a selector 444.

The estimated mixture ratio processing portion 443 calculates theestimated mixture ratio of the indicated pixel belonging to theuncovered background area by using calculation by equation (22) on thebasis of the pixel value input from the selector 441, and supplies thecalculated estimated mixture ratio to a selector 444.

When the target pixel belongs to the foreground area, the selector 444selects the estimated mixture ratio of 0 on the basis of the areainformation supplied from the area specifying unit 103, and sets it tothe mixture ratio α and when the target pixel belongs to the backgroundarea, the selector 444 selects the estimated mixture ratio of 1 and setsit to the mixture ratio α. When the target pixel belongs to the coveredbackground area, the selector 444 selects the estimated mixture ratiosupplied from the estimated mixture ratio processor 442 to set it to themixture ratio α. When the target pixel belongs to the uncoveredbackground area, the selector 444 selects the estimated mixture ratiosupplied from the estimated mixture ratio processor 443 to set it to themixture ratio α. The selector 444 outputs the mixture ratio α selectedand set on the basis of the area information.

Like the above, the mixture ratio calculator 104 having anotherconfiguration shown in FIG. 80 can calculate the mixture ratio α foreach pixel having an image and output the calculated mixture ratio α.

Referring to a flowchart of FIG. 81, the mixture ratio α calculatingprocess of the mixture ratio calculator 104 of which configuration isshown in FIG. 72 will be explained. In step S401, the mixture ratiocalculator 104 obtains the area information supplied from the areaspecifying unit 103. In step S402, the estimated mixture ratio processor401 performs the estimated mixture ratio calculating process by using amodel corresponding to the covered background area and supplies thecalculated estimated mixture ratio to the mixture ratio determiningportion 403. Details of the estimated mixture ratio calculating processwill be explained later referring to a flowchart of FIG. 82.

In step S403, the estimated mixture ratio processor 402 performs theestimated mixture ratio calculating process by using a modelcorresponding to the uncovered background area and supplies thecalculated estimated mixture ratio to the mixture ratio determiningportion 403.

In step S404, the mixture ratio calculator 104 determines whether themixture ratios α are estimated for the whole frame or not, and if it isdetermined that the mixture ratios α are not estimated for the wholeframes, the procedure returns to step S402 and performs processing ofestimating the mixture ratio α for next pixel.

In step S404, if it is determined that the mixture ratio α is estimatedfor the whole frame, the procedure proceeds to step S405 and the mixtureratio determining portion 403 sets the mixture ratio α on the basis ofthe area information supplied from the area specifying unit 103,indicating which of the foreground area, the background area, thecovered background area and the uncovered background area the pixelbelongs to. The mixture ratio determining portion 403 sets 0 to themixture ratio α when the target pixel belongs to the foreground area,sets 1 to the mixture ratio α when the target pixel belongs to thebackground area, sets the estimated mixture ratio supplied from theestimated mixture ratio processor 401 to the mixture ratio α when thetarget pixel belongs to the covered background area and sets theestimated mixture ratio supplied from the estimated mixture ratioprocessor 402 to the mixture ratio α when the target pixel belongs tothe uncovered background area. Then, the process ends.

Like the above, the mixture ratio calculator 104 can calculate themixture ratio α that is a feature amount corresponding to each pixel, onthe basis of the area information supplied from the area specifying unit103 and the input image.

The mixture ratio α calculation processing of the mixture ratiocalculator 104 of which configuration is shown in FIG. 80 is similar tothat explained in the flowchart of FIG. 81 and thus, the explanationthereof will be omitted.

Next, a process of estimating a mixture ratio by using a modelcorresponding to the covered background area, corresponding to step S402in FIG. 81, will be explained referring to the flowchart of FIG. 82.

In step S421, the mixture ratio calculator 423 obtains the pixel value Cof the indicated pixel in the frame #n from the frame memory 421.

In step S422, the mixture ratio calculator 423 obtains the pixel value Pof pixel in the frame #n−1 corresponding to the indicated pixel from theframe memory 422.

In step S423, the mixture ratio calculator 423 obtains the pixel value Nof pixel in the frame #n+1 corresponding to the indicated pixel includedin the input image.

In step S424, the mixture ratio calculator 423 calculates the estimatedmixture ratio on the basis of the pixel value C of the indicated pixelin the frame #n, the pixel value P of pixel in the frame #n−1 and thepixel value N of pixel in the frame #n+1.

In step S425, the mixture ratio calculator 423 determines whether theestimated mixture ratio calculation processing is finished for the wholeframe or not and if it is determined that the estimated mixture ratiocalculation processing is not finished for the whole frame, theprocedure returns to step S421 and repeats the estimated mixture ratiocalculation processing for next pixel.

In step S425, if it is determined that the estimated mixture ratiocalculating processing is finished for the whole frame, the processends.

Like this, the estimated mixture ratio processor 401 can calculate theestimated mixture ratio on the basis of the input image.

The mixture ratio estimating process by using a model corresponding tothe uncovered background area in step S403 in FIG. 81 is similar to theprocess shown in the flowchart of FIG. 82, in which the both process useequation corresponding to the model of the uncovered background area andthus, explanation thereof will be omitted.

Further, since the estimated mixture ratio processor 442 and theestimated mixture ratio processor 443 shown in FIG. 80 perform the sameprocessing as that in the flowchart of FIG. 82 to calculate theestimated mixture ratio, explanation thereof will be omitted.

Further, although it has been explained that the object corresponding tothe background is stationary, even when the image corresponding to thebackground area includes motion, the mixture ratio α calculating processmentioned above can be applied. For example, when the imagecorresponding to the background area is constantly moving, the estimatedmixture ratio processor 401 shifts the whole image according to motionof the background, and performs the same processing as in the case wherethe object corresponding to the background is stationary. Further, whenthe image corresponding to the background area includes local motion ofother background, the estimated mixture ratio processor 401 selects thepixels corresponding to the motion of the background as the pixelscorresponding to the pixels belonging to the mixed area, and performsthe aforementioned process.

Further, the mixture ratio calculator 104 may perform the mixture ratioestimating processing for all the pixels by only using a modelcorresponding to the covered background area, and may output thecalculated estimated mixture ratio as the mixture ratio α. In this case,the mixture ratio α means the ratio of the background components for thepixels belonging to the covered background area and the ratio of theforeground components for the pixels belonging to the uncoveredbackground area. If the absolute value of the difference between themixture ratio α is calculated like the above method and 1 for the pixelsbelonging to the uncovered background area and the calculated absolutevalue is set to the mixture ratio α, the separation server 11 can obtainthe mixture ratio α indicating the ratio of the background componentsfor the pixels belonging to the uncovered background area.

Further, similarly, the mixture ratio calculator 104 may perform onlythe mixture ratio estimating processing for all the pixels by using amodel corresponding to the uncovered background area, and may output thecalculated estimated mixture ratio as the mixture ratio α.

Next, the mixture ratio calculator 104 for calculating the mixture ratioα by using characteristics that the mixture ratio α is varied linearlywill be explained.

As described above, since each of equation (11) and equation (12)includes two variables, the mixture ratio α cannot be obtained fromthese equations.

Therefore, equation approximating the mixture ratio α and sum f of theforeground components in the spatial direction is established by usingcharacteristics that the mixture ratio α is varied linearlycorresponding to the positional variation of pixels according tomovement of the foreground object at a uniform speed for the shuttertime. The equation approximating the mixture ratio α and sum f of theforeground components is solved by using the pixel value of pixelbelonging to the mixed area and the pixel value of pixel belonging tothe background area.

If variation of the mixture ratio α is linearly approximated, themixture ratio α is expressed by equation (23).

α=il+p   (23)

In this equation (23), i is an index in the spatial direction when theposition of the indicated pixel is 0. l is slope of a linear line of themixture ratio α. p is an intercept of the linear line of the mixtureratio α and is the mixture ratio α of the indicated pixel. In equation(23), the index i is known but the slope l and the intercept p are notknown.

The relationship between index i, the slope l and the intercept p isshown in FIG. 83.

By approximating the mixture ratio α by equation (23), a plurality ofdifferent mixture ratios α for a plurality of pixels are expressed withtwo variables. In the example shown in FIG. 83, 5 mixture ratios for thefive pixels are expressed with two variables of the slope i and theintercept p. Further, in FIG. 83, the indicated pixel is indicated by awhite circle and the peripheral pixels are indicated by black circle.

If the mixture ratio α is approximated in a plane shown in FIG. 84,considering movement v corresponding to two directions, the horizontaldirection and the vertical direction of the image, the equation (23) isextended into the plane and the mixture ratio α is expressed by equation(24). Further, in FIG. 84, the white circle indicates the indicatedpixel.

α=jm+kq+p   (24)

In equation (24), j is an index in the horizontal direction when theposition of the indicated pixel is 0 and k is an index in the verticaldirection. m is a slope of the mixture ratio α plane in the horizontaldirection and q is a slope of the mixture ratio α plane in the verticaldirection. p is an intercept of the mixture ratio α plane.

For example, in the frame #n shown in FIG. 74, equations (25) to (27)are established with respect to C05 to C07, respectively.

C05=α05·B05/v+f05   (25)

C06=α06·B06/v+f06   (26)

C07=α07·B07/v+f07   (27)

If the foreground components are equal in the neighborhood, that is, F01to F03 are equal and F01 to F03 is replaced with Fc, equation (28) canbe established.

f(x)=(1−α(x))·Fc   (28)

In equation (28), x expresses a position of the spatial direction.

If α (x) is replaced with equation (24), and equation (28) can beexpressed by equation (29).

$\begin{matrix}\begin{matrix}{{f(x)} = {\left( {1 - \left( {{jm} + {kq} + p} \right)} \right) \cdot {Fc}}} \\{= {{j \cdot \left( {{- m} \cdot {Fc}} \right)} + {k \cdot \left( {{- q} \cdot {Fc}} \right)} + \left( {\left( {1 - p} \right) \cdot {Fc}} \right)}} \\{= {{js} + {kt} + u}}\end{matrix} & (29)\end{matrix}$

In equation (29), (−m·Fc), (−q·Fc) and (1−p)·Fc are replaced as shown inequations (30) to (32).

s=−m·Fc   (30)

t=−q·Fc   (31)

u=(1−p)·Fc   (32)

In equation (29), j is an index in the horizontal direction whenposition of the indicated pixel is 0 and k is an index in the verticaldirection.

Like the above, since it is supposed that the object corresponding tothe foreground is moving at a uniform speed for the shutter time and thecomponents corresponding to the foreground are constant in theneighborhood, the sum of the foreground components is approximated byequation (29).

Further, when the mixture ratio α is approximated by a linear line, thesum of the foreground components can be expressed by equation (33).

f(x)=is+u   (33)

If substituting the mixture ratio α in equation (13) and the sum of theforeground components in equation (24) and equation (29), the pixelvalue M is expressed by equation (34).

$\begin{matrix}\begin{matrix}{M = {{\left( {{jm} + {kq} + p} \right) \cdot B} + {js} + {kt} + u}} \\{= {{{jB} \cdot m} + {{kB} \cdot q} + {B \cdot p} + {j \cdot s} + {k \cdot t} + u}}\end{matrix} & (34)\end{matrix}$

In equation (34), the six unknown variables are the slope m of themixture ratio α plane in the horizontal direction, the slope q of themixture ratio α plane in the vertical direction and the intercepts p, s,t and u of the mixture ratio α plane.

The pixel value M or the pixel value B is substituted in a normalequation shown in equation (34) corresponding to pixels in theneighborhood of the indicated pixel and a plurality of normal equationsin which the pixel value M or the pixel value B is substituted aresolved by the least square method to calculate the mixture ratio α.

For example, if the index j of the indicated pixel in the horizontaldirection is 0, the index k in the vertical direction is 0 and the pixelvalue M or the pixel value B is substituted in the normal equation shownin equation (34) for 3×3 pixels adjacent to the indicated pixel,equations (35) to (43) are obtained.

M _(−1, −1)=(−1)·B _(−1, −1) ·m+(−1)·B _(−1, −1) ·q+B _(−1, −1)·p+(−1)·s+(−1)·t+u   (35)

M _(0, −1)=(0)·B _(0, −1) ·m+(−1)·B _(0, −1) ·q+B _(0, −1) ·p+(−1)·t+u  (36)

M _(+1, −1)=(−1)·B _(+1, −1) ·m+(−1)·B _(+1, −1) ·q+B _(−1, −1)·p+(+1)·s+(−1)·t+u   (37)

M _(−1, 0)=(−1)·B _(−1, 0) ·m+(0)·B _(−1, 0) ·q+B _(−1, 0)·p+(−1)·s+(0)·t+u   (38)

M _(0, 0)=(0)·B _(0, 0) ·m+(0)·B _(0, 0) ·q+B _(0, 0) ·p+(0)·s+(0)·t+u  (39)

M _(+1, 0)=(+1)·B _(−1, 0) ·m+(0)·B _(+1, 0) ·q+B _(+1, 0)·p+(+1)·s+(0)·t+u   (40)

M _(−1, +1)=(−1)·B _(−1, +1) ·m+(+1)·B _(−1, +1) ·p+(−1)·s+(+1)·+u  (41)

M _(0, +1)=(0)·B _(0, +1) ·m+(+1)·B _(0, +1) ·q+B _(0, +1)·p+(0)·s+(+1)·t+u   (42)

M _(+1, +1)=(+1)·B _(+1, +1) ·m+(+1)·B _(+1, +1) ·q+B _(+1, +1)·p+(+1)·s+(+1)·t+u   (43)

Since the index j of the indicated pixel in the horizontal direction is0 and the index k in the vertical direction is 0, the mixture ratio α ofthe indicated pixel is equal to the value when j=0 and k=0 in equation(24), that is, the intercept p.

Therefore, each value of the slope m in the horizontal direction, theslope q in the vertical direction and the intercepts p, s, t and u iscalculated by using the least square method on the basis the nineequations (35) to (43) and then, the intercept p is output as themixture ratio α.

Next, a more specific procedure for calculating the mixture ratio α byusing the least square method will be explained.

If the index i and the index k are expressed by one index x, therelationship of the index i, the index k and the index x is expressed byequation (44).

x=(j+1)·3+(k+1)   (44)

The slope m in the horizontal direction, the slope q in the verticaldirection and the intercepts p, s, t, and u are expressed as variablesw0, w1, w2, w3, w4 and w5, respectively and jB, kB, B, j, k and l areexpressed as a0, a1, a2, a3, a4 and a5, respectively. Considering errorex, equations (35) to (43) can be rewritten as equation (45).

$\begin{matrix}{M_{x} = {{\sum\limits_{y = 0}^{5}\; {a_{y} \cdot w_{y}}} + {e_{x}\;}}} & (45)\end{matrix}$

In equation (45), x is a value from integers 0 to 8.

Equation (46) can be derived from the equation (45).

$\begin{matrix}{e_{x} = {M_{x} - {\sum\limits_{y = 0}^{5}\; {a_{y} \cdot w_{y}}}}} & (46)\end{matrix}$

Here, in order to use the least square method, the sum of squares oferrors E is defined as shown by equation (47).

$\begin{matrix}{E = {\sum\limits_{x = 0}^{8}\; e_{x^{2}}}} & (47)\end{matrix}$

In order to minimize error, the partial differential of variable Wv andthe sum of squares of errors E should be 0. Here, v is one of integers 0to 5. Therefore, wy is obtained to satisfy equation (48).

$\begin{matrix}\begin{matrix}{\frac{\partial E}{\partial w_{v}} = {2 \cdot {\sum\limits_{x = 0}^{8}\; {e_{x} \cdot \frac{\partial e_{x}}{\partial w_{v}}}}}} \\{= {{2 \cdot {\sum\limits_{x = 0}^{8}\; {e_{x} \cdot a_{v}}}} = 0}}\end{matrix} & (48)\end{matrix}$

If equation (46) is substituted in equation (48), equation (49) isobtained.

$\begin{matrix}{{\sum\limits_{x = 0}^{8}\left( {a_{v} \cdot {\sum\limits_{y = 0}^{5}{a_{y} \cdot w_{y}}}} \right)} = {\sum\limits_{x = 0}^{8}{a_{v} \cdot M_{x}}}} & (49)\end{matrix}$

By applying, for example, sweeping-out method such as Gauss-Jordanelimination method, to six equations obtained by substituting one ofintegers 0 to 5 in v of equation (49), wy is calculated. As describedabove, w0 is the slope m in the horizontal direction, w1 is the slope inthe vertical direction, w2 is the intercept p, w3 is s, w4 is t and w5is u.

As described above, by applying the least square method to equations inwhich the pixel value M and the pixel value B are substituted, the slopem in the horizontal direction, the slope q in the vertical direction andthe intercepts p, s, t and u can be obtained.

In explanation of equations (35) to (43), the pixel value of pixelincluded in the mixed area is explained as M and the pixel value ofpixel included in the background area is explained as B. However, it isnecessary that normal equations be established for each case that theindicated pixel is included in the covered background area or in theuncovered background area.

For example, when the mixture ratios α of pixels included in the coveredbackground area in the frame #n as shown in FIG. 74 are obtained, thepixel values C04 to C08 of pixels in the frame #n and the pixel valuesP04 to P08 of pixels in the frame #n−1 are substituted in the normalequations.

When the mixture ratios α of pixels included in the uncovered backgroundarea in the frame #n as shown in FIG. 75 are obtained, the pixel valuesC28 to C32 of pixels in the frame #n and the pixel values N28 to N32 ofpixels in the frame #n+1 are substituted in the normal equations.

Further, for example, when the mixture ratios α of pixels included inthe covered background area in the frame #n as shown in FIG. 85 arecalculated, equations (50) to (58) can be established. The pixel valueof pixel of which the mixture ratio α is calculated is Mc5. Further, inFIG. 85, white circles indicate the pixels in the background and blackcircles indicate the pixels in the mixed area, respectively.

Mc1=(−1)·Bc1·m+(−1)·Bc1·q+Bc1·p+(−1)·s+(−1)·t+u   (50)

Mc2=(0)·Bc2·m+(−1)·Bc2·q+Bc2·p+(0)·s+(−1)·t+u   (51)

Mc3=(+1)·Bc3·m+(−1)·Bc3·q+Bc3·p+(+1)·s+(−1)·t+u   (52)

Mc4=(−1)·Bc4·m+(0)·Bc4·q+Bc4·p+(−1)·s+(0)·t+u   (53)

Mc5=(0)·Bc5·m+(0)·Bc5·q+Bc5·p+(0)·s+(0)·t+u   (54)

Mc6=(+1)·Bc6·m+(0)·Bc6·q+Bc6·p+(+1)·s+(0)·t+u   (55)

Mc7=(−1)·Bc7·m+(+1)·Bc7·q+Bc7·p+(−1)·s+(+1)·t+u   (56)

Mc8=(0)·Bc8·m+(+1)·Bc8·q+Bc8·p+(0)·s+(+1)·t+u   (57)

Mc9=(+1)·Bc9·m+(+1)·Bc9·q+Bc9·p+(+1)·s+(+1)·t+u   (58)

When the mixture ratios α of pixels included in the covered backgroundarea in the frame #n are calculated, the pixel values Bc1 to Bc9 ofpixels of the background area in the frame #n−1 corresponding to thepixels in the frame #n are used in equations (50) to (58).

When the mixture ratios α of pixels included in the uncovered backgroundshown in FIG. 85 are calculated, equations (59) to (67) can beestablished. The pixel value of pixel of which the mixture ratio α iscalculated is Mu5.

Mu1=(−1)·Bu1·m+(−1)·Bu1·q+Bu1·p+(−1)·s+(−1)·t+u   (59)

Mu2=(0)·Bu2·m+(−1)·Bu2·q+Bu2·p+(0)·s+(−1)·t+u   (60)

Mu3=(+1)·Bu3·m+(−1)·Bu3·q+Bu3·p+(+1)·s+(−1)·t+u   (61)

Mu4=(−1)·Bu4·m+(0)·Bu4·q+Bu4·p+(−1)·s+(0)·t+u   (62)

Mu5=(0)·Bu5·m+(0)·Bu5·q+Bu5·p+(0)·s+(0)·t+u   (63)

Mu6=(+1)·Bu6·m+(0)·Bu6·q+Bu6·p+(+1)·s+(0)·t+u   (64)

Mu7=(−1)·Bu7·m+(+1)·Bu7·q+Bu7·p+(−1)·s+(+1)·t+u   (65)

Mu8=(0)·Bu8·m+(+1)·Bu8·q+Bu8·p+(0)·s+(+1)·t+u   (66)

Mu9=(+1)·Bu9·m+(+1)·Bu9·q+Bu9·p+(+1)·s+(+1)·t+u   (67)

When the mixture ratios α of pixels included in the uncovered backgroundarea in the frame #n are calculated, the pixel values Bu1 to Bu9 ofpixels of the background area in the frame #n+1 corresponding to thepixels in the frame #n are used in equations (59) to (67).

FIG. 86 is a block diagram illustrating configuration of the estimatedmixture ratio processor 401. The image input to the estimated mixtureratio processor 401 is supplied to a delay circuit 501 and an adder 502.

The delay circuit 501 delays the input image by one, frame and suppliesit to the adder 502. When the frame #n is input to the adder 502 as theinput image, the delay circuit 501 supplies the frame #n−1 to the adder502.

The adder 502 sets the pixel values of pixels adjacent to pixel of whichthe mixture ratio α is calculated and the pixel values in the frame #n−1in the normal equations. For example, the adder 502 sets the pixelvalues Mc1 to Mc9 and the pixel values Bc1 to Bc9 in the normalequations on the basis of equations (50) to (58). The adder 502 suppliesthe normal equations in which the pixel values are set to the calculator503.

The calculator 503 solves the normal equations supplied from the adder502 by using the elimination method, etc. to obtain the estimatedmixture ratio and outputs the obtained estimated mixture ratio.

Like the above, the estimated mixture ratio processor 401 can calculatethe estimated mixture ratio on the basis of the input image and supplyit to the mixture ratio determining portion 403.

Further, since the estimated mixture ratio processor 402 has theconfiguration similar to that of the estimated mixture ratio processor401, explanation thereof will be omitted.

FIG. 87 shows an example of the estimated mixture ratio calculated bythe estimated mixture ratio processor 401. The estimated mixture ratioshown in FIG. 87 expresses the result calculated from equationsgenerated in a unit of 7×7 pixels with respect to one line when theamount of movement v of the foreground corresponding to the objectmoving at a uniform speed is 11.

It can be known that the estimated mixture ratio is varied almostlinearly in the mixed area as shown in FIG. 86.

Next, the mixture ratio estimating processing of the estimated mixtureratio processor 401 of which configuration is shown in FIG. 86 by usinga model corresponding to the covered background area will be explainedreferring to FIG. 88.

In step S521, the adder 502 sets the pixel values included in the inputimage and the pixel values included in the image supplied from the delaycircuit 501 in the normal equations corresponding to the model of thecovered background area.

In step S522, the estimated mixture ratio processor 401 determineswhether setting for the target pixels is finished or not and if it isdetermined that the setting for the target pixels is not finished, theprocedure returns to step S521 and repeats the setting processing of thepixel values in the normal equations.

In step S522, if it is determined that the setting the pixel value forthe target pixels is finished, the procedure proceeds to step S523 andthe calculator 173 calculates the estimated mixture ratio on the basisof the normal equations in which the pixel values are set and outputsthe obtained estimated mixture ratio.

Like the above, the estimated mixture ratio processor 401 of whichconfiguration is shown in FIG. 86 can calculate the estimated mixtureratio on the basis of the input image.

The mixture ratio estimating process using the model corresponding tothe uncovered background area is similar to the process using the normalequations corresponding to the model of the uncovered background areashown in a flowchart of FIG. 88 and thus, explanation thereof will beomitted.

Further, although it has been explained that the object corresponding tothe background is stationary, even when the image corresponding to thebackground area includes movement, the aforementioned mixture ratioobtaining processing can be applied. For example, when the imagecorresponding to the background is moving constantly, the estimatedmixture ratio processor 401 shifts the whole image according to themovements and then, processes similarly to the case that the objectcorresponding to the background is stationary. Further, when the imagecorresponding to the background area includes other local movements, theestimated mixture ratio processing portion 401 selects the pixelscorresponding to the movements as the pixels corresponding to pixelsbelonging to the mixed area and then, performs the aforementionedprocessing.

Like the above, the mixture ratio calculator 102 can calculate themixture ratio α that is a feature amount corresponding to each pixel onthe basis of the area information supplied from the area specifying unit101 and the input image.

By using the mixture ratio α, while the information of motion blurincluded in the image corresponding to the moving object is left, it ispossible to separate the foreground components and the backgroundcomponents included in the pixel values.

Further, if an image is synthesized on the basis of the mixture ratio α,it is possible to generate image including the corrected motion bluradapted to the speed of the object moving to correspond to the images inthe real world.

Next, the foreground/background separator 105 will be explained. FIG. 89is a block diagram illustrating an example of a configuration of theforeground/background separator 105. The input image supplied to theforeground/background separator 105 is supplied to a separating portion601, the switch 602 and the switch 604. The area information indicatingthe covered background area and the area information indicating theuncovered background area supplied from the area specifying unit 103,are supplied to the separating portion 601. The area informationindicating the foreground area is supplied to the switch 602. The areainformation indicating the background area is supplied to the switch604.

The mixture ratio α supplied from the mixture ratio calculator 104 issupplied to the separating portion 601.

The separating portion 601 separates the foreground components from theinput image to supply the separated foreground components to thesynthesizer 603 and separates the background components from the inputimage to supply the separated background components to the synthesizer605, on the basis of the area information indicating the coveredbackground area, the area information indicating the uncoveredbackground area and the mixture ratio α.

When the pixels corresponding to the foreground are input, the switch602 is closed and supplies only the pixels corresponding to theforeground included in the input image to the synthesizer 603, on thebasis of the area information indicating the foreground area.

When the pixels corresponding to the background are input, the switch604 is closed and supplies only the pixels corresponding to thebackground included in the input image to the synthesizer 605, on thebasis of the area information indicating the background area.

The synthesizer 603 synthesizes the foreground component image on thebasis of the foreground components supplied from the separating portion601 and the pixels corresponding to the foreground supplied from theswitch 602, and outputs the synthesized foreground component image.Since the foreground area and the background area are not overlapped,the synthesizer 603 applies, for example, logical sum operation to theforeground components and the pixels corresponding to the foreground tosynthesize the foreground component image.

In initial process to be first performed in the process of synthesizingthe foreground component image, the synthesizer 603 stores the image ofwhich all the pixels in the built-in frame memory are 0, and in theprocess of synthesizing the foreground component image, the synthesizer603 stores (overwrites) the foreground component image. Therefore, 0 isstored as the pixel values in the pixels corresponding to the backgroundarea and in the foreground component images supplied from thesynthesizer 603.

The synthesizer 605 synthesizes the background component image on thebasis of the background components supplied from the separating portion601 and the pixels corresponding to the background supplied from, theswitch 604, and outputs the synthesized background component image.Since the background area and the mixed area are not overlapped, thesynthesizer 605 applies, for example, logical sum operation to thebackground components and the pixels corresponding to the background tosynthesize the background component image.

In initial process to be first performed in the process of synthesizingthe background component image, the synthesizer 605 stores the image ofwhich all the pixels in the built-in frame memory are 0, and in theprocess of synthesizing the background component image, the synthesizer605 stores (overwrites) the background component image. Therefore, 0 isstored as the pixel values in the pixels corresponding to the foregroundarea and in the background component images supplied from thesynthesizer 605.

FIGS. 90A and 90B show the input image input to theforeground/background separator 105 and the foreground component imageand background component image output from the foreground/backgroundseparator 105.

FIG. 90A shows a typical diagram of an image to be displayed, and FIG.90B shows a model diagram in which the pixels in one line includingpixels belonging to the foreground area, pixels belonging to thebackground area and pixels belonging to the mixed area corresponding toFIG. 90A are expanded in the time direction.

As shown in FIGS. 90A and 90B, the background component image outputfrom the foreground/background separator 105 comprise the pixelsbelonging to the background area and the background components includedin the pixels in the mixed area.

As shown in FIGS. 90A and 90B, the foreground component image outputfrom the foreground/background separator 105 comprise the pixelsbelonging to the foreground area and the foreground components includedin the pixels in the mixed area.

The pixel values of pixels in the mixed area are separated into thebackground components and the foreground components by theforeground/background separator 105. The separated background componentsconstitute the background component image along with the pixelsbelonging to the background area. The separated foreground componentsconstitute the foreground component image along with the pixelsbelonging to the foreground area.

Like the above, in the foreground component image, the pixel values ofpixels corresponding to the background area are 0, and significant pixelvalues are set to the pixels corresponding to the foreground area andthe pixels corresponding to the mixed area. Similarly, in the backgroundcomponent image, the pixel values of pixels corresponding to theforeground area are 0, and significant pixel values are set to thepixels corresponding to the background area and the pixels correspondingto the mixed area.

Next, a process of separating the foreground components and thebackground components from the pixels belonging to the mixed area, to beperformed by the separating portion 601 will be explained.

FIG. 91 is a model diagram of an image including the foregroundcorresponding to an object moving from the left to right in the drawingand indicating the foreground components and the background componentsin two frames. In the model of image shown in FIG. 91, the amount ofmovement v of the foreground is defined as 4 and the virtual divisionnumber is set at 4.

In a frame #n, the leftmost pixel and the fourteenth to eighteenthpixels from the leftmost include only the background components andbelong to the background area. In the frame #n, the second to fourthpixels from the leftmost include the background components and theforeground components and belong to the uncovered background area. Inthe frame #n, the eleventh to thirteenth pixels from the leftmostinclude the background components and the foreground components andbelong to the covered background area. In the frame #n, the fifth totenth pixels from the leftmost include only the foreground componentsand belong to the foreground area.

In a frame #n+1, the first to fifth pixels from the leftmost and theeighteenth pixel from the leftmost include only the backgroundcomponents and belong to the background area. In the frame #n+1, thesixth to eighth pixels from the leftmost include the backgroundcomponents and the foreground components and belong to the uncoveredbackground area. In the frame #n+1, the fifteenth to seventeenth pixelsfrom the leftmost include the background components and the foregroundcomponents and belong to the covered background area. In the frame #n+1,the ninth to fourteenth pixels from the leftmost include only theforeground components and belong to the foreground area.

FIG. 92 is a diagram illustrating a process of separating the foregroundcomponents from the pixels belonging to the covered background area. InFIGS. 92, α1 to α18 are the mixture ratios corresponding to therespective pixels in the frame #n. In FIG. 92, the fifteenth toseventeenth pixels from the leftmost belong to the covered backgroundarea.

The pixel value C15 of the fifteenth pixel from the leftmost in theframe #n is expressed by equation (68).

$\begin{matrix}\begin{matrix}{{C\; 15} = {{B\; {15/v}} + {F\; {09/v}} + {F\; {08/v}} + {F\; {07/v}}}} \\{= {{\alpha \; {15 \cdot B}\; 15} + {F\; {09/v}} + {F\; {08/v}} + {F\; {07/v}}}} \\{= {{\alpha \; {15 \cdot P}\; 15} + {F\; {09/v}} + {F\; {08/v}} + {F\; {07/v}}}}\end{matrix} & (68)\end{matrix}$

Here, α15 is a mixture ratio of the fifteenth pixel from the leftmost inthe frame #n. P15 is a pixel value of the fifteenth pixel from theleftmost in the frame #n−1.

On the basis of equation (68), sum f15 of the foreground components ofthe fifteenth pixel from the leftmost in the frame #n is expressed byequation (69).

$\begin{matrix}\begin{matrix}{{f\; 15} = {{F\; {09/v}} + {F\; {08/v}} + {F\; {07/v}}}} \\{= {{C\; 15} - {\alpha \; {15 \cdot P}\; 15}}}\end{matrix} & (69)\end{matrix}$

Similarly, sum f16 of the foreground components of the sixteenth pixelfrom the leftmost in the frame #n is expressed by equation (70), and sumf17 of the foreground components of the seventeenth pixel from theleftmost in the frame #n is expressed by equation (71).

f16=C16−α16·P16   (70)

f17=C17−α17·P17   (71)

Like the above, the foreground component fc included in the pixel valueC of pixels belonging to the covered background area is calculated byequation (72).

fc=C−α·P   (72)

P is the pixel value of the corresponding pixel in the frame one ahead.

FIG. 93 is a diagram illustrating a process of separating the foregroundcomponents from the pixels belonging to the uncovered background area.In FIGS. 93, α1 to α18 are the mixture ratios corresponding to therespective pixels in the frame #n. In FIG. 93, the second to fourthpixels from the leftmost belong to the uncovered background area.

The pixel value C02 of the second pixel from the leftmost in the frame#n is expressed by equation (73).

$\begin{matrix}\begin{matrix}{{C\; 02} = {{B\; {02/v}} + {B\; {02/v}} + {B\; {02/v}} + {F\; {01/v}}}} \\{= {{\alpha \; {2 \cdot B}\; 02} + {F\; {01/v}}}} \\{= {{\alpha \; {2 \cdot N}\; 02} + {F\; {01/v}}}}\end{matrix} & (73)\end{matrix}$

Here, α2 is the mixture ratio of the second pixel from the leftmost inthe frame #n. N02 is the pixel value of the second pixel from theleftmost in the frame #n+1.

On the basis of equation (73), sum f02 of the foreground components ofthe second pixel from the leftmost in the frame #n is expressed byequation (74).

$\begin{matrix}\begin{matrix}{{f\; 02} = {F\; {01/v}}} \\{= {{C\; 02} - {\alpha \; {2 \cdot N}\; 02}}}\end{matrix} & (74)\end{matrix}$

Similarly, sum f03 of the foreground components of the third pixel fromthe leftmost in the frame #n is expressed by equation (75) and sum f04of the foreground components of the fourth pixel from the leftmost inthe frame #n is expressed by equation (76)

f03=C03−α3·N03   (75)

f04=C04−α4·N04   (76)

Like this, the foreground component fu included in the pixel value C ofpixel belonging to the uncovered background area is calculated byequation (77).

fu=C−α  (77)

N is the pixel value of the corresponding pixel in the frame one behind.

Like the above, the separating portion 601 can separate the foregroundcomponents and the background components from the pixel belonging to themixed area on the basis of the information indicating the coveredbackground area, the information indicating the uncovered backgroundarea which are included in the area information and the mixture ratio αfor each pixel.

FIG. 94 is a block diagram illustrating an example of configuration ofthe separating portion 601 to perform the aforementioned processing. Theimage input to the separating portion 601 is supplied to the framememory 621 and the area information indicating the covered backgroundarea and the uncovered background area supplied from the mixture ratiocalculator 104 and the mixture ratio α are input to the separationprocessing block 622.

The frame memory 621 stores the input image in frame unit. When thetarget of processing is the frame #n, the frame memory 621 stores theframe #n−1 one ahead of the frame #n the frame #n and the frame #n+1 onebehind the frame #n.

The frame memory 621 supplies pixels corresponding to the frame #n−1,the frame #n and the frame #n+1 to the separation processing block 622.

The separation processing block 622 separates the foreground componentand background component from the pixel belonging to the mixed area ofthe frame #n on the basis of the area information indicating the coveredbackground area and the uncovered background area and the mixture ratioα by applying the pixel values of the pixels corresponding to the frame#n−1, the frame #n and the frame #n+1, supplied from the frame memory621 by the calculation explained in FIGS. 92 and 93 and supplies them tothe frame memory 623.

The separation processing block 622 comprises an uncovered-areaprocessor 631, a covered-area processor 632, a synthesizer 633 andsynthesizer 634.

A multiplier 641 of the uncovered-area processor 631 multiplies thepixel value of the pixel in the frame #n+1 supplied from the framememory 621 by the mixture ratio α and outputs the result to the switch642. The switch 642 is closed when the pixel in the frame #n suppliedfrom the frame memory 621 (corresponding to the pixel of the frame #n+1)belongs to the uncovered background area, and supplies the valueobtained by multiplying the pixel value by the mixture ratio α suppliedfrom the multiplier 641 to the calculator 643 and the synthesizer 634.The value obtained by multiplying the pixel value of the pixel in theframe #n+1 output from the switch 642 by the mixture ratio α equals tothe background component of the pixel value of the pixel correspondingto the frame #n.

The calculator 643 obtains the foreground component by subtracting thebackground component which is supplied from the switch 642 from thepixel value of the pixel of the frame #n which is supplied from theframe memory 621, and supplies the foreground component of the pixel ofthe frame #n belonging to the uncovered background area to thesynthesizer 633.

The multiplier 651 of the covered-area processor 632 multiplies thepixel value of the pixel of the frame #n−1 supplied from the framememory 621 by the mixture α, and outputs the result to the switch 652.The switch 652 is closed when the pixel of frame #n which is suppliedfrom the frame memory 621 (corresponding to the pixel of the frame #n−1)belongs to the covered background area, and supplies the value obtainedby multiplying the pixel value by the mixture ratio α which is suppliedfrom the multiplier 651 to the calculator 653 and the synthesizer 634.The value obtained by multiplying the pixel value of the pixel of theframe #n−1 output from the switch 652 by the mixture ratio α equals tothe background component of the pixel value of the pixel correspondingto the frame #n.

The calculator 653 obtains the foreground component by subtracting thebackground component which is supplied from switch 652 from the pixelvalue of the pixel of the frame #n which is supplied from the framememory 621, and supplies the foreground component of the pixel of theframe #n belonging to the covered background area to the synthesizer633.

The synthesizer 633 synthesizes the foreground component of the pixel ofthe frame #n, belonging to the uncovered background area and suppliedfrom the calculator 643 and the foreground component of the pixel of theframe #n, belonging to the covered background area and supplied from thecalculator 653, and then supplies the result to the frame memory 623.

The synthesizer 634 synthesizes the background component of the pixel ofthe frame #n, belonging to the uncovered background area and suppliedfrom the switch 642 and the background component of the pixel of theframe #n, belonging to the covered background area and supplied from thecalculator 652, and then supplies the result to the frame memory 623.

The frame memory 623 stores respectively the foreground component andthe background component of the pixels of the mixed area of the frame #nwhich are supplied from the separation processing block 622.

The frame memory 623 outputs the stored foreground component andbackground component of the pixels of the mixed area of the frame #n.

It is possible to perfectly separate the foreground component from thebackground component which are included in the pixel value by using themixture ratio α which is a feature amount.

The synthesizer 603 synthesizes the foreground component of the pixel ofthe mixed area of frame #n output from the separating portion 601 andthe pixel belongs to the foreground area to generate the foregroundcomponent image. The synthesizer 605 synthesizes the backgroundcomponent of the pixel of the mixed area of frame #n output from theseparating portion 601 and the pixel belongs to the background area togenerate the background component image.

FIGS. 95A and 95B are drawings showing an example of the foregroundcomponent image and an example of the background component imagecorresponding to the frame #n of FIG. 91.

FIG. 95A shows an example of the foreground component imagecorresponding to the frame #n of FIG. 91. Because the leftmost pixel andthe fourteenth pixel from the leftmost comprise only the backgroundcomponent before they are separated into the foreground and background,their respective pixel values become 0.

The second to fourth pixels from the leftmost belong to the uncoveredbackground area before they are separated into the foreground andbackground. Therefore, their respective background components become 0and their respective foreground components remain unchanged. Theeleventh to thirteenth pixels from the leftmost belong to the coveredbackground area before they are separated into the foreground andbackground. Therefore, their respective background components become 0and their respective foreground components remain unchanged. Because thefifth to tenth pixel from the leftmost comprise only the foregroundcomponent, their respective pixel values remain unchanged.

FIG. 95B shows an example of the background component imagecorresponding to the frame #n of FIG. 91. Because the leftmost pixel andthe fourteenth pixel from the leftmost comprise only the backgroundcomponent before they are separated into the foreground and background,their respective pixel values remain unchanged.

The second to fourth pixels from the leftmost belong to the uncoveredbackground area before they are separated into the foreground andbackground. Therefore, their respective foreground components become 0and their respective background components remain unchanged. Theeleventh to thirteenth pixels from the leftmost belong to the coveredbackground area before they are separated into the foreground andbackground. Therefore, their respective foreground components become 0and their respective background components remain unchanged. Because thefifth to tenth pixel from the leftmost comprise only the foregroundcomponent, their respective pixel values become 0.

Next, referring to a flowchart in FIG. 96, a process of separating theforeground and the background by the foreground/background separator 105will be explained. In step S601, the frame memory 621 of the separatingportion 601 obtains the input image, and stores the indicated frame #ntargeted to be separated into the foreground and background with theleading frame #n−1 and the following frame #n+1.

In step S602, the separation processing block 622 of the separatingportion 601 obtains the area information supplied from the mixture ratiocalculator 104. In step S603, the separation processing block 622 of theseparating portion 601 obtains the mixture ratio α supplied from themixture ratio calculator 104.

In step S604, the uncovered-area processor 631 extracts the backgroundcomponent from the pixel value of the pixel belonging to the uncoveredbackground area supplied from the frame memory 621 on the basis of thearea information and the mixture ratio α.

In step S605, the uncovered area processor 631 extracts the foregroundcomponent from the pixel value of the pixel belonging to the uncoveredbackground area supplied from the frame memory 621 on the basis of thearea information and the mixture ratio α.

In step S606, the covered area processor 632 extracts the backgroundcomponent from the pixel value of the pixel belonging to the coveredbackground area supplied from the frame memory 621 on the basis of thearea information and the mixture ratio α.

In step S607, the uncovered area processor 632 extracts the foregroundcomponent from the pixel value of the pixel belonging to the coveredbackground area supplied from the frame memory 621 on the basis of thearea information and the mixture ratio α.

In step S608, the synthesizer 633 synthesizes the foreground componentof the pixel belonging to the uncovered background area and extractedfrom the processing in step S605 and the foreground component of thepixel belonging to the covered background area and extracted from theprocessing in step S607. The synthesized foreground component issupplied to the synthesizer 603. Further, the synthesizer 603synthesizes the pixel belonging to the foreground area and suppliedthrough the switch 602 and the foreground component supplied from theseparating portion 601, and then generates the foreground componentimage.

In step S609, the synthesizer 634 synthesizes the background componentof the pixel belonging to the uncovered background area and extractedfrom the processing in step S604 and the background component of thepixel belonging to the covered background area and extracted from theprocessing in step S606. The synthesized background component issupplied to the synthesizer 605. Further, the synthesizer 605synthesizes the pixel belonging to the background area and suppliedthrough the switch 604 and the background component supplied from theseparating portion 601, and then generates the foreground componentimage.

In step S610, the synthesizer 603 outputs the foreground componentimage. In step S611, the synthesizer 605 outputs the backgroundcomponent image and then the process ends.

Like the above, the foreground/background separator 105 can separate theforeground component and the background component from the input imagebased on the area information and the mixture ratio α and output theforeground component image comprised of only the foreground componentand the background component image comprised of only the backgroundcomponent.

Next, an adjustment process of adjusting the amount of motion blur ofthe foreground component image is explained.

FIG. 97 is a block diagram showing an example of a structure of themotion blur adjusting unit 106. The motion vector supplied from themotion detector 102 and the positional information thereof are suppliedto a unit of processing determining portion 801, a model forming portion802 and a calculator 805. The area information supplied from the areaspecifying unit 103 is supplied to the unit of processing determiningportion 801. The foreground component image supplied from theforeground/background separator 105 is supplied to an adder 804.

The unit of processing determining portion 801 generates the unit ofprocessing based on the motion vector, the positional informationthereof and the area information, and then supplies the generated unitof processing to the model forming portion 802 and the adder 804.

The unit of processing A generated by the unit of processing determiningportion 801, as shown in FIG. 98, indicates the successive pixels whichare in a row in the moving direction from the pixel corresponding to thecovered background area of the foreground component image to the pixelcorresponding to the uncovered background area, or the successive pixelswhich are in a row in the moving direction from the pixel correspondingto the uncovered background area to the pixel corresponding to thecovered background area. The unit of processing A, for example,comprises data of left-upper point (position of the pixel which isdesignated as one of the unit of processing and is located at theleftmost or the most upper on the image) and data of right-lower point.

The model forming portion 802 performs the model forming based on themotion vector and the input unit of processing A. More specifically, themodel forming portion 802 stores, for example, a plurality of modelscorresponding to the number of pixels included in the unit of processingA, the virtual division number of pixel value in time direction and thenumber of the foreground components for each pixel in advance, and thenselects a designated model corresponding to the pixel value and theforeground component shown in FIG. 99 on the basis of the unit ofprocessing A and the virtual division number of the pixel value in timedirection.

For example, if the number of pixels of the unit of processing A is 12and the amount of movement v in shutter time is 5, the model formingportion 802 selects a model totally comprised of 8 foreground componentsand where the virtual division number is 5, the leftmost pixel includes1 foreground component, the second pixel from the leftmost includes 2foreground components, the third pixel from the leftmost includes 3foreground components, the fourth pixel from the leftmost includes 4foreground components, the fifth pixel from the leftmost includes 5foreground components, the sixth pixel from the leftmost includes 5foreground components, the seventh pixel from the left includes 5foreground components, the eighth pixel from the leftmost includes 5foreground components, the ninth pixel from the leftmost includes 4foreground components, the tenth pixel from the leftmost includes 3foreground components, the eleventh pixel from the leftmost includes 2foreground components, the twelfth pixel from the leftmost includes 1foreground component.

Further, the model forming portion 802 may not select one of the modelsstored in advance, but may generate a model based on motion vector andthe unit of processing when the motion vector and the unit of processingA are supplied.

The model forming portion 802 supplies the selected model to theequation generator 803.

The equation generator 803 generates equations based on the modelssupplied from the model forming portion 802. Referring to the models ofthe foreground component image in FIG. 99, when the number of theforeground components is 8, the number of the pixels corresponding tothe unit of processing A is 12, the amount of movement v is 5 and thevirtual division number is 5, subsequent equations generated by theequation generator 803 are explained.

When the foreground component included in the foreground component imagecorresponding to the shutter time/v lies between F01/v and F08/v, therelations between F01/v to F08/v and the pixel values C01 to C12 areexpressed in equations (78) to (89).

C01=F01/v   (78)

C02=F02/v+F01/v   (79)

C03=F03/v+F02/v+F01/v   (80)

C04=F04/v+F03/v+F02/v+F01/v   (81)

C05=F05/v+F04/v+F03/v+F02/v+F01/v   (82)

C06=F06/v+F05/v+F04/v+F03/v+F02/v   (83)

C07=F07/v+F06/v+F05/v+F04/v+F03/v   (84)

C08=F08/v+F07/v+F06/v+F05/v+F04/v   (85)

C09=F08/v+F07/v+F06/v+F05/v   (86)

C10=F08/v+F07/v+F06/v   (87)

C11=F08/v+F07/v   (88)

C12=F08/v   (89)

The equation generator 803 generates equations by modifying thegenerated equations. Equations generated by the equation generator 803are expressed in equations (90) to (101).

$\begin{matrix}{{C\; 01} = {{{1 \cdot F}\; {01/v}} + {{0 \cdot F}\; {02/v}} + {{0 \cdot F}\; {03/v}} + {{0 \cdot F}\; {04/v}} + {{0 \cdot F}\; {05/v}} + {{0 \cdot F}\; {06/v}} + {{0 \cdot F}\; {07/v}} + {{0 \cdot F}\; {08/v}}}} & (90) \\{{C\; 02} = {{{1 \cdot F}\; {01/v}} + {{1 \cdot F}\; {02/v}} + {{0 \cdot F}\; {03/v}} + {{0 \cdot F}\; {04/v}} + {{0 \cdot F}\; {05/v}} + {{0 \cdot F}\; {06/v}} + {{0 \cdot F}\; {07/v}} + {{0 \cdot F}\; {08/v}}}} & (91) \\{{C\; 03} = {{{1 \cdot F}\; {01/v}} + {{1 \cdot F}\; {02/v}} + {{1 \cdot F}\; {03/v}} + {{0 \cdot F}\; {04/v}} + {{0 \cdot F}\; {05/v}} + {{0 \cdot F}\; {06/v}} + {{0 \cdot F}\; {07/v}} + {{0 \cdot F}\; {08/v}}}} & (92) \\{{C\; 04} = {{{1 \cdot F}\; {01/v}} + {{1 \cdot F}\; {02/v}} + {{1 \cdot F}\; {03/v}} + {{1 \cdot F}\; {04/v}} + {{0 \cdot F}\; {05/v}} + {{0 \cdot F}\; {06/v}} + {{0 \cdot F}\; {07/v}} + {{0 \cdot F}\; {08/v}}}} & (93) \\{{C\; 05} = {{{1 \cdot F}\; {01/v}} + {{1 \cdot F}\; {02/v}} + {{1 \cdot F}\; {03/v}} + {{1 \cdot F}\; {04/v}} + {{1 \cdot F}\; {05/v}} + {{0 \cdot F}\; {06/v}} + {{0 \cdot F}\; {07/v}} + {{0 \cdot F}\; {08/v}}}} & (94) \\{{C\; 06} = {{{0 \cdot F}\; {01/v}} + {{1 \cdot F}\; {02/v}} + {{1 \cdot F}\; {03/v}} + {{1 \cdot F}\; {04/v}} + {{1 \cdot F}\; {05/v}} + {{1 \cdot F}\; {06/v}} + {{0 \cdot F}\; {07/v}} + {{0 \cdot F}\; {08/v}}}} & (95) \\{{C\; 07} = {{{0 \cdot F}\; {01/v}} + {{0 \cdot F}\; {02/v}} + {{1 \cdot F}\; {03/v}} + {{1 \cdot F}\; {04/v}} + {{1 \cdot F}\; {05/v}} + {{1 \cdot F}\; {06/v}} + {{1 \cdot F}\; {07/v}} + {{0 \cdot F}\; {08/v}}}} & (96) \\{{C\; 08} = {{{0 \cdot F}\; {01/v}} + {{0 \cdot F}\; {02/v}} + {{0 \cdot F}\; {03/v}} + {{1 \cdot F}\; {04/v}} + {{1 \cdot F}\; {05/v}} + {{1 \cdot F}\; {06/v}} + {{1 \cdot F}\; {07/v}} + {{1 \cdot F}\; {08/v}}}} & (97) \\{{C\; 09} = {{{0 \cdot F}\; {01/v}} + {{0 \cdot F}\; {02/v}} + {{0 \cdot F}\; {03/v}} + {{0 \cdot F}\; {04/v}} + {{1 \cdot F}\; {05/v}} + {{1 \cdot F}\; {06/v}} + {{1 \cdot F}\; {07/v}} + {{1 \cdot F}\; {08/v}}}} & (98) \\{{C\; 10} = {{{0 \cdot F}\; {01/v}} + {{0 \cdot F}\; {02/v}} + {{0 \cdot F}\; {03/v}} + {{0 \cdot F}\; {04/v}} + {{0 \cdot F}\; {05/v}} + {{1 \cdot F}\; {06/v}} + {{1 \cdot F}\; {07/v}} + {{1 \cdot F}\; {08/v}}}} & (99) \\{{C\; 11} = {{{0 \cdot F}\; {01/v}} + {{0 \cdot F}\; {02/v}} + {{0 \cdot F}\; {03/v}} + {{0 \cdot F}\; {04/v}} + {{0 \cdot F}\; {05/v}} + {{0 \cdot F}\; {06/v}} + {{1 \cdot F}\; {07/v}} + {{1 \cdot F}\; {08/v}}}} & (100) \\{{C\; 12} = {{{0 \cdot F}\; {01/v}} + {{0 \cdot F}\; {02/v}} + {{0 \cdot F}\; {03/v}} + {{0 \cdot F}\; {04/v}} + {{0 \cdot F}\; {05/v}} + {{0 \cdot F}\; {06/v}} + {{0 \cdot F}\; {07/v}} + {{1 \cdot F}\; {08/v}}}} & (101)\end{matrix}$

Equations (90) to (102) may be expressed as an equation (102).

$\begin{matrix}{C_{j} = {\sum\limits_{i = 01}^{08}{{aij} \cdot {F_{i}/v}}}} & (102)\end{matrix}$

In equation (102), j means the position of pixel. In this example, j isone of 1 to 12. Further, i means the position of foreground value. Inthis example, i is one of 1 to 8. aij is 0 or 1 according to the valuesof i and j.

Considering an error, equation (102) is expressed as equation (103).

$\begin{matrix}{{Cj} = {{\sum\limits_{i = 01}^{08}{{aij} \cdot {F_{i}/v}}} + {ej}}} & (103)\end{matrix}$

In equation (103), ej is an error included in indicated pixel Cj.

Equation 103 can be expressed as equation 104 again.

$\begin{matrix}{{ej} = {{Cj} - {\sum\limits_{i = 01}^{08}{{aij} \cdot {F_{i}/v}}}}} & (104)\end{matrix}$

Here, sum of the square of errors E is defined as shown in equation(105) to apply the least square method.

$\begin{matrix}{E = {\sum\limits_{j = 01}^{12}{ej}^{2}}} & (105)\end{matrix}$

In order to minimize the error, the value of the partial differential bythe variable Fk to the sum of the square of errors E should equal zero.Fk satisfying equation (106) is obtained.

$\begin{matrix}\begin{matrix}{\frac{\partial E}{\partial F_{k}} = {2 \cdot {\sum\limits_{j = 01}^{12}{{ej} \cdot \frac{\partial{ej}}{\partial F_{k}}}}}} \\{= {2 \cdot {\sum\limits_{j = 01}^{12}\left\{ {{\left( {{Cj} - {\sum\limits_{i = 01}^{08}{{aij} \cdot {F_{i}/v}}}} \right) \cdot \left( {{- a_{kj}}/v} \right)} = 0} \right.}}}\end{matrix} & (106)\end{matrix}$

In equation (106), because the amount of movement v is fixed, equation(107) can be derived.

$\begin{matrix}{{\sum\limits_{j = 01}^{12}{a_{kj} \cdot \left( {{Cj} - {\sum\limits_{i = 01}^{08}{{aij} \cdot {F_{i}/v}}}} \right)}} = 0} & (107)\end{matrix}$

Equation (108) is obtained by expanding and transposing equation (107).

$\begin{matrix}{{\sum\limits_{j = 01}^{12}\left( {a_{kj} \cdot {\sum\limits_{i = 01}^{08}{{aij} \cdot F_{i}}}} \right)} = {v \cdot {\sum\limits_{j = 01}^{12}{a_{kj} \cdot {Cj}}}}} & (108)\end{matrix}$

Equation (108) is expanded in 8 equations obtained by substituting oneof integers between 1 and 8 for k in equation (108). The obtained 8equations can be expressed as a matrix equation, and this matrixequation is called a normal equation.

An example of a normal equation generated by the equation generator 803based on the least square method is shown in equation (109).

$\begin{matrix}{{\begin{bmatrix}5 & 4 & 3 & 2 & 1 & 0 & 0 & 0 \\4 & 5 & 4 & 3 & 2 & 1 & 0 & 0 \\3 & 4 & 5 & 4 & 3 & 2 & 1 & 0 \\2 & 3 & 4 & 5 & 4 & 3 & 2 & 1 \\1 & 2 & 3 & 4 & 5 & 4 & 3 & 2 \\0 & 1 & 2 & 3 & 4 & 5 & 4 & 3 \\0 & 0 & 1 & 2 & 3 & 4 & 5 & 4 \\0 & 0 & 0 & 1 & 2 & 3 & 4 & 5\end{bmatrix}\begin{bmatrix}{F\; 01} \\{F\; 02} \\{F\; 03} \\{F\; 04} \\{F\; 05} \\{F\; 06} \\{F\; 07} \\{F\; 08}\end{bmatrix}} = {v \cdot \begin{bmatrix}{\sum\limits_{i = 08}^{12}C_{i}} \\{\sum\limits_{i = 07}^{11}C_{i}} \\{\sum\limits_{i = 06}^{10}C_{i}} \\{\sum\limits_{i = 05}^{09}C_{i}} \\{\sum\limits_{i = 04}^{08}C_{i}} \\{\sum\limits_{i = 03}^{07}C_{i}} \\{\sum\limits_{i = 02}^{06}C_{i}} \\{\sum\limits_{i = 01}^{05}C_{i}}\end{bmatrix}}} & (109)\end{matrix}$

If equation (109) is expressed as A·F=v·C, C, A and v are known and F isunknown. A and v are known at the time of model forming and C is alreadyknown by inputting the pixel values in adding operation.

Errors included in pixel C can be dispersed by calculating theforeground component by using the normal equation based on the leastsquare method.

The equation generator 803 supplies the normal equation generated by theabove to the adder 804.

The adder 804 sets the pixel values C included in foreground componentimage in the matrix equation supplied from the equation generator 803 onthe basis of the unit of processing supplied from the unit of processingdetermining portion 801. The adder 804 supplies the matrix where thepixel values C are set to the calculator 805.

The calculator 805 calculates the foreground component Fi/v from whichmotion blur is eliminated through the processing based on thesweeping-out methods such as Gauss-Jordan elimination method, calculatesFi corresponding to the pixel value i of the foreground from whichmotion blur is eliminated, wherein i is one of integers between 0 and 8,and then outputs the foreground component image from which motion bluris eliminated, along with the calculated pixel value Fi to the motionblur adder 806 and the selector 807.

Further, as shown in FIG. 100, for making the position of the foregroundcomponent image unchanged with respect to the screen, each of F01 to F08is set to each of C03 to C10 in the foreground component image fromwhich motion blur is eliminated. However, it may correspond to arbitrarypositions.

The motion blur adder 806 may adjust the amount of motion blur by addingthe amount by which motion blur is adjusted v′ which is different fromthe amount of movement v, for example, the amount by which motion bluris adjusted v′ is half of the amount of movement v, or has no relationwith the amount of movement v. For example, as shown in FIG. 101, themotion blur adder 806 calculates the foreground component Fi/v′ bydividing the foreground pixel value from which motion blur is eliminatedFi with the amount by which motion blur is adjusted v′, calculates thesum of the foreground components Fi/v′, and then generates the pixelvalue where the amount of motion blur is adjusted. For example, when theamount by which motion blur is adjusted is 3, the pixel value C02becomes (F01)/v′, the pixel value C03 becomes (F01+F02)/v′, the pixelvalue C04 becomes (F01+F02+F03)/v′ and the pixel value C05 becomes(F02+F03+F04)/v′.

The motion blur adder 806 supplies the foreground component image ofwhich the amount of motion blur is adjusted to the selector 807.

The selector 807 selects one of the foreground component image fromwhich motion blur is eliminated and which is supplied from thecalculator 805 and the foreground component image of which the amount ofmotion blur is adjusted and which is supplied from the motion blur adder806 on the basis of the selection signal selected by an user, and thenoutputs the selected foreground component image.

Like the above, the motion blur adjusting unit 106 is able to adjust theamount of motion blur based on the selection signal and the amount v′ bywhich motion blur is adjusted.

Further, for example, as shown in FIG. 102, when the number of pixelscorresponding to the unit of processing is 8 and the amount of movementv is 4, the motion blur adjusting unit 106 generates the matrix equationshown in equation (110).

$\begin{matrix}{{\begin{bmatrix}4 & 3 & 2 & 1 & 0 \\3 & 4 & 3 & 2 & 1 \\2 & 3 & 4 & 3 & 2 \\1 & 2 & 3 & 4 & 3 \\0 & 1 & 2 & 3 & 4\end{bmatrix}\begin{bmatrix}{F\; 01} \\{F\; 02} \\{F\; 03} \\{F\; 04} \\{F\; 05}\end{bmatrix}} = {v \cdot \begin{bmatrix}{\sum\limits_{i = 05}^{08}C_{i}} \\{\sum\limits_{i = 04}^{07}C_{i}} \\{\sum\limits_{i = 03}^{06}C_{i}} \\{\sum\limits_{i = 02}^{05}C_{i}} \\{\sum\limits_{i = 01}^{04}C_{i}}\end{bmatrix}}} & (110)\end{matrix}$

The motion blur adjusting unit 106 generates as many equations needed tocorrespond to the length of the unit of processing and calculates thepixel value Fi from which the amount of motion blur is adjusted. In thesame manner, when the number of pixels included in the unit ofprocessing is, for example, 100, the motion blur adjusting unit 106generates as many equations to accommodate the unit of processingcomprising one hundred pixels, and then calculates Fi.

FIG. 103 is a drawing showing another structure of the motion bluradjusting unit 106. The same portions as those in FIG. 97 are given withthe same reference numerals and the description thereof will be omitted.

The selector 821 supplies the input motion vector and the positionalinformation thereof, as it is, to the unit of processing determiningportion 801 and the model forming portion 802, on the basis of theselection signal or replaces the magnitude of motion vector with theamount v′ by which motion blur is adjusted and then supplies thereplaced amount v′ and the positional information thereof to the unit ofprocessing determining portion 801 and the model forming portion 802.

By doing like this, the unit of processing determining portion 801 tothe calculator 805 of the motion blur adjusting unit 106 in FIG. 103 canadjust the amount of motion blur according to the amount of movement vand the amount v′ by which motion blur is adjusted. For example, whenthe amount of movement v is 5 and the amount by which motion blur isadjusted is 3, the unit of processing determining portion 801 to thecalculator 805 of the motion blur adjusting unit 106 in FIG. 103performs calculations for the foreground component image whose amount ofmovement v is 5 according to the model of FIG. 101 corresponding to theamount v′ by which the motion blur is adjusted 3, and then calculatedthe image including the motion blur according to the (amount of movementv)/(amount v′ by which motion blur is adjusted)=5/3, almost equaling1.7. Further, in this case, because the calculated image does notinclude the motion blur corresponding to amount of movement v of 3, itis important to keep in mind that the result of the motion blur adder806 is different from the relation between the amount of movement v andthe amount of adjusting motion blur v′.

As mentioned above, the motion blur adjusting unit 106 generatesequations, according to the amount of movement v and the unit ofprocessing, sets the pixel values of foreground component image in thegenerated equations, and then calculates the foreground component imagewhose amount of motion blur is adjusted.

Next, referring a flow chart in FIG. 104, a process of adjusting ofamount of motion blur included in foreground component image by themotion blur adjusting portion 106 is explained.

In step S801, the unit of processing determining portion 801 of themotion blur adjusting unit 106 generates the unit of processing based onthe motion vector and the positional information thereof, and thesupplies the generated unit of processing to the model forming portion802.

In step S802, the model forming portion 802 of the motion blur adjustingunit 106 selects or generates a model according to the amount ofmovement v and the unit of processing. In step S803, the equationgenerator 803 generates a normal equation based on the selected model.

In step S804, the adder 804 sets the pixel values of the foregroundcomponent image in the generated normal equation. In step S805, theadder 804 determines whether the pixel values of all the pixelscorresponding to the unit of processing are set or not and if it isdetermined that the pixel value of all the pixels corresponding to theunit of processing are not set, the procedure returns to step S804 andrepeats the processing for setting the pixel values in the generatednormal equation.

In step S805, if the adder 804 determines that the pixel values of allthe pixels corresponding to the unit of processing are set, theprocedure proceeds to set S806, the calculator 805 calculates the pixelvalues of foreground whose amount of motion blur is adjusted on thebasis of the normal equation where the pixel values supplied by theadder 804 is set, and the process ends.

Like this, the motion blur adjusting unit 106 can adjust the amount ofmotion blur from foreground image which includes the motion blur basedon the motion vector and the positional information thereof.

That is, it is possible to adjust the amount of motion blur included inthe pixel values which are sample data.

FIG. 105 is a block diagram showing another example of a configurationof the motion blur adjusting unit 106. The motion vector and thepositional information thereof supplied from the motion detector 102 aresupplied to a unit of processing determining portion 901 and a adjustingportion 905, and the area information supplied from the area specifyingunit 103 is supplied to the unit of processing determining portion 901.The foreground component image supplied from the foreground/backgroundseparator 105 is supplied to calculator 904.

The unit of processing determining portion 901 generates a unit ofprocessing based on the motion vector and positional informationthereof, and the area information, and then supplies the generated unitof processing to the model forming portion 902 along with the motionvector.

The model forming portion 902 performs the model forming based on themotion vector and the input unit of processing. More specifically, themodel forming portion 902 stores, for example, a plurality of modelscorresponding to the number of pixels included in the unit ofprocessing, the virtual division number of the pixel value in timedirection and the number of the foreground components for each pixel inadvance, selects a specifying model corresponding to the pixel value andthe foreground component shown in FIG. 106 on the basis of the unit ofprocessing and the virtual division number of the pixel value in timedirection.

For example, if the number of pixels corresponding to the unit ofprocessing is 12 and the amount of movement v is 5, the model formingportion 902 selects the model, totally comprised 8 foreground componentsand where the number of division is 5, the leftmost pixel includes oneforeground component, the second pixel from the leftmost includes twoforeground components, the third pixel from the leftmost includes threeforeground components, the fourth pixel from the leftmost includes fourforeground components, the fifth pixel from the leftmost includes fiveforeground components, the sixth pixel from the leftmost includes fiveforeground components, the seventh pixel from the leftmost includes fiveforeground components, the eighth pixel from the leftmost includes fiveforeground components, the ninth pixel from the leftmost includes fourforeground components, the tenth pixel from the leftmost includes threeforeground components, the eleventh pixel from the leftmost includes twoforeground components, the twelfth pixel from the leftmost includes oneforeground component.

Further, the model forming portion 902 may not select one of the modelsstored in advance, but generates a model based on the motion vector andthe unit of processing when the motion vector and the unit of processingare supplied.

The equation generator 903 generates equations based on the modelsupplied from the model forming portion 902.

Referring to models of the foreground component image shown in FIGS. 106to 108, examples of the equations generated by the equation generator903 is explained when the number of the foreground components is 8, thenumber of the pixels corresponding to the unit of processing is 12 andthe amount of movement v is 5.

When the foreground component corresponding to (shutter time)/v includedin the foreground component image is between F01/v and F08/v, therelationship between F01/v to F08/v and the pixel values C01 to C12 areexpressed in equations (78) to (89) as mentioned earlier.

Paying attention to the pixel values C12 and C11, the pixel value C12comprise only the foreground component F08/v as shown in equation (111)and the pixel value C11 comprise the sum of the foreground componentsF08/v and F07/v. Therefore, the foreground component F07/v can beobtained from equation (112).

F08/v=C12   (111)

F07/v=C11−C12   (112)

In the same manner, referring to the foreground components included inthe pixel values C10 to C01, the foreground components F06/v to F01/vcan be obtained by the equations (113) to (118).

F06/v=C10−C11   (113)

F05/v=C09−C10   (114)

F04/v=C08−C09   (115)

F03/v=C07−C08+C12   (116)

F02/v=C06−C07+C11−C12   (117)

F01/v=C05−C06+C10−C11   (118)

The equation generator 903 generates equations for calculating theforeground components by using the differences between the pixel valuesshown in equations (111) to (118). The equation generator 903 suppliesthe generated equations to the calculator 904.

The calculator 904 calculates the foreground components based on theequations supplied from the equation generator 903 where the pixelvalues of the foreground component image are set. For example, when theequations (111) to (118) are supplied from the equation generator 903,the pixel values C05 to C12 are substituted in the equations (111) to(118).

The calculator 904 calculates the foreground components based on theequations where the pixel values are set. For example, the calculator904 calculates the foreground components F01/v to F08/v from thecalculations based on the equations (111) to (118) where the pixelvalues C05 to C12 are set as shown in FIG. 107. The calculator 904supplies the foreground components F01/v to F08/v to the adjustingportion 905.

The adjusting portion 905 calculates the pixel value of the foregroundwhere motion blur is eliminated by multiplying the foreground componentsupplied from the calculator 904 by the amount of movement v included inmotion vector supplied from the unit of processing determining portion901. For example, the adjusting portion 905 calculates the pixel valuesF01 to F08 of the foregrounds where motion blur is eliminated bymultiplying each of the foreground components F01/v to F08/v by 5 of theamount of movement v when the foreground components F01/v to F08/vsupplied from the calculator 904 are supplied as shown in FIG. 108.

The adjusting portion 905 supplies the calculated foreground componentimage comprising the pixel values of the foreground where motion blur iseliminated to the motion blur adder 906 and the selector 907.

The motion blur adder 906 may adjust the amount of motion blur by, forexample, the amount v′ by which motion blur is adjusted which isdifferent from the amount of movement v and which is half of the amountof movement v, or the amount v′ by which motion blur is adjusted havingno relation with the amount of movement v. For example, as shown in FIG.101, the motion blur adder 906 calculates the foreground component Fi/v′by eliminating the pixel value of foreground from which motion blur iseliminated Fi from the amount v′ by which motion blur is adjusted,calculates the sum of the foreground components Fi/v′, and thengenerates the pixel value where the amount of motion blur is adjusted.For example, when the amount by which motion blur is adjusted is 3, thepixel value C02 becomes (F01)/v′, the pixel value C03 becomes(F01+F02)/v′, the pixel value C04 becomes (F01+F02+F03)/v′ and the pixelvalue C05 becomes (F02+F03+F04)/v′.

The motion blur adder 906 supplies the foreground component image wheremotion blur is adjusted to the selector 907.

The selector 907 selects one of the foreground component image fromwhich motion blur is eliminated and which is supplied from the adjustingportion 905, and the foreground component image of which motion blur isadjusted and which is supplied from the motion blur adder 906, forexample, on the basis of selection signal corresponding to the selectionof an user, and then outputs the selected foreground component image.

Like the above, the motion blur adjusting unit 106 can adjust the amountof motion blur based on the selection signal and the amount v′ by whichmotion blur is adjusted.

Next, referring the flow chart in FIG. 109, a process of adjusting theamount of motion blur of foreground in the motion blur adjusting unit106 having a configuration shown in FIG. 105, will be explained.

In step S901, the unit of processing determining portion 901 of themotion blur adjusting unit 106 generates a unit of processing based onthe motion vector and the area information, and then supplies thegenerated unit of processing to the model forming portion 902 and theadjusting portion 905.

In step S902, the model forming portion 902 of the motion blur adjustingunit 106 selects or generates a model according to the amount ofmovement v and the unit of processing. In step S903, the equationgenerator 903 generates an equation for calculating the foregroundcomponent from the differences between the pixel values of theforeground components based on the selected or generated model.

In step S904, the calculator 904 sets the pixel values of the foregroundcomponent image in the generated equation and extracts the foregroundcomponent from the differences between the pixel values on the basis ofthe equation which sets the pixels. In step S905, the calculator 904determines whether all the foreground components corresponding to theunit of processing are extracted, and if it is determined that all theforeground components corresponding to the unit of processing are notextracted, the procedure returns to step S904 and the processing forextracting the foreground components is repeated.

In step S905, if it is determined that all the foreground componentscorresponding to unit of processing are extracted, the procedureproceeds to step S906, and the adjusting portion 905 corrects each ofthe foreground components F01/v to F08/v supplied from the calculator904 based on the amount v of movement and then calculates the pixelvalues F01 to F08 from which the motion blur is eliminated.

In step S907, the motion blur adder 906 calculates the pixel values ofthe foreground where amount of motion blur is adjusted, the selector 907selects either the image in which amount of movement is eliminated orthe image in which amount of movement is adjusted and outputs theselected image, and then the procedure is finished.

Like the above, the motion blur adjusting unit 106 having theconfiguration shown in FIG. 105 can adjust motion blur by foregroundimage in which motion blur is included more rapidly by more simpleoperations.

Conventional methods for eliminating motion blur such as wiener•filter,etc., are effective in ideal conditions, but are not effective for thequantized real images having noises. However, this method using themotion blur adjusting unit 106 having the structure shown in FIG. 105 issufficiently effective for the quantized real images having noises andit is possible to eliminate the motion blur precisely.

As mentioned above, the separation server 11 having the structure shownin FIG. 27 can adjust the amount of motion blur belonging to the inputimage.

FIG. 110 is a block diagram showing another configuration of theseparation server 11.

The same portions as those in FIG. 27 are given with the same referencenumerals and the description thereof will be omitted adequately.

The area specifying unit 103 supplies the area information to themixture ratio calculator 104 and the synthesizer 1001.

The mixture ratio calculator 104 supplies the mixture ratio α to theforeground/background separator 105 and the synthesizer 1001.

The foreground/background separator 105 the foreground component imageto the synthesizer 1001.

The synthesizer 1001 synthesizes any background image and the foregroundcomponent image supplied from the foreground/background separator 105based on the mixture ratio α supplied from the mixture ratio calculator104 and the area information supplied from the area specifying unit 103,and then output the synthesized image corresponding to the backgroundimage or the foreground component image.

FIG. 111 is a drawing showing a configuration of the synthesizer 1001.The background component generator 1021 generates the backgroundcomponent image based on the mixture ratio α and any background image,and then supplies the generated background component image to the mixedarea image synthesizing portion 1022.

The mixed area image synthesizing portion 1022 generates the mixed areasynthesized image by synthesizing the background component imagesupplied from the background component generator 1021 and the foregroundcomponent and then supplies the generated mixed area synthesized imageto the image synthesizing portion 1023.

The image synthesizing portion 1023 synthesizes the foreground componentimage, the mixed area synthesized image supplied from the mixed areaimage synthesizing portion 1022 and any background image based on thearea information to generate and output the synthesized image.

Like the above, the synthesizer 1001 can synthesize the foregroundcomponent image and any background image.

The image obtained by synthesizing the foreground component image andany background image based on the feature amount of the mixture ratio αis more natural when compared to the image obtained by synthesizing onlythe pixels.

FIG. 112 is a block diagram showing another configuration of theseparation server 11 which has adjusted the amount of motion blur. Theseparation server 11 shown in FIG. 27 performs the area specificationand calculates of the mixture ratio α serially, while the separationserver 11 shown in FIG. 112 performs the area specification andcalculates of the mixture ratio α in parallel.

The same portions as those in FIG. 27 are given with the same referencenumerals and the description thereof will be omitted.

The input image is supplied to the mixture ratio calculator 1101, theforeground/background separator 1102, the area specifying unit 103 andthe object extracting unit 101.

The mixture ratio calculator 1101 calculates an estimated mixture ratiosupposing that the pixel belongs to the covered background area andcalculates an estimated mixture ratio supposing that the pixel belongsto the uncovered background area for each pixel belonging to the inputimage on the basis of the input image, and supplies the estimatedmixture ratio for the case supposing that the pixel belongs to thecovered background area and the estimated mixture ratio in the case thatsupposes the pixel belongs to the uncovered background area to theforeground/background separator 1102.

FIG. 113 is a block diagram showing an example of a configuration of theestimated mixture ratio calculator 1101.

An estimated mixture ratio processor 401 shown in FIG. 113 is the sameas the estimated mixture ratio processor 401 shown in FIG. 72. Anestimated mixture ratio processor 402 shown in FIG. 113 is the same asthe estimated mixture ratio processor 402 shown in FIG. 72.

The estimated mixture ratio processor 401 calculates an estimatedmixture ratio for each pixel using the calculation corresponding to amodel of the covered background area based on the input image, and thenoutputs the calculated estimated mixture ratio.

The estimated mixture ratio processor 402 calculates an estimatedmixture ratio for each pixel using the calculation corresponding to amodel of the uncovered background area based on the input image, andthen outputs the calculated estimated mixture ratio.

The foreground/background separator 1102 generates a foregroundcomponent image from the input image, based on the estimated mixtureratio for the case that supposes the pixel belongs to the coveredbackground area, the estimated mixture ratio for the case that supposesthe pixel belongs to the uncovered background area which are suppliedfrom the mixture ratio calculator 1101, and based on the areainformation supplied from the area specifying unit 103, and thensupplies the generated foreground component image to the motion bluradjusting unit 106 and the selector 107.

FIG. 114 is a block diagram showing an example of a configuration theforeground/background separator 1102.

The same portions as those in the foreground/background separator 105shown in FIG. 89 are given with the same reference numerals and thedescription thereof will be omitted.

The selector 1121 selects the estimated mixture ratio for the case thatsupposes the pixel belongs to the covered background area or theestimated mixture ratio for the case that supposes the pixel belongs tothe uncovered background area which are supplied from the mixture ratiocalculator 1101, based on the area information supplied from the areaspecifying unit 103, and then supplies the selected estimated mixtureratio as the mixture ratio α to the separating portion 601.

The separating portion 601 extracts the foreground component andbackground component from the pixel values of the pixel belonging to themixed area based on the mixture ratio and the area information suppliedfrom the selector 1121, supplies the extracted foreground component tothe synthesizer 603 and supplies the extracted background component tothe synthesizer 605.

The separating portion 601 can have the same configuration shown in FIG.94.

The synthesizer 603 synthesizes and outputs the foreground componentimage. The synthesizer 605 synthesizes and outputs the backgroundcomponent image.

The motion blur adjusting unit 106 shown in FIG. 112, which may have thesame structure as the configuration shown in FIG. 27, adjusts the amountof motion blur belonging to the foreground component image supplied fromthe foreground/background separator 1102 based on the area informationand the motion vector, and then outputs the foreground component imagewhere the amount of motion blur is adjusted.

The selector 107 shown in FIG. 112 selects either the foregroundcomponent image supplied from the foreground/background separator 1102or the foreground component image in which the motion blur is adjustedsupplied from the motion blur adjusting unit 106 based on the selectionsignal corresponding the selection of a user, and then output theselected foreground component image.

Like the above, the separation server 11 having the configuration shownin FIG. 112 can adjust the amount of motion blur included in the imagecorresponding to the foreground object included in the input image andoutputs it. The separation server 11 having the configuration shown inFIG. 112 can calculate a mixture ratio α of buried information andoutput the calculated mixture ratio α as the same manner in the firstembodiment.

FIG. 115 is a block diagram showing another configuration of theseparation server 11 for synthesizing the foreground component image andany background image. The separation server 11 shown in FIG. 110performs the area specification and the calculation of the mixture ratioα serially. But the separation server 11 shown in FIG. 115 performs thearea specification and the calculation of the mixture ratio α inparallel.

The same portions as those in FIG. 112 are given with the same referencenumerals and the description thereof will be omitted.

The mixture ratio calculator 1101 shown in FIG. 115 calculates anestimated mixture ratio for the case that supposes the pixel belongs tothe covered background area and an estimated mixture ratio for the casethat supposes the pixel belongs to the uncovered background area foreach pixel belonging to the input image on the basis of the input image,and supplies the estimated mixture ratio for the case that supposes thepixel belongs to the covered background area and the estimated mixtureratio for the case that supposes the pixel belongs to the uncoveredbackground area to the foreground/background separator 1102 and thesynthesizer 1201.

The foreground/background separator 1102 shown in FIG. 115 generates aforeground component image from the input image based on the estimatedmixture ratio for the case that supposes the pixel belongs to thecovered background area and the estimated mixture ratio for the casethat supposes the pixel belongs to the uncovered background area whichare supplied from the mixture ratio calculator 1101, and based on thearea information supplied from the area specifying unit 103, and thensupplies the generated foreground component image to the synthesizer1201.

The synthesizer 1201 synthesizes any background image and the foregroundcomponent image supplied from the foreground/background separator 1102based on the estimated mixture ratio for the case that supposes thepixel belongs to the covered background area and the estimated mixtureratio for the case that supposes the pixel belongs to the uncoveredbackground area which are supplied from the mixture ratio calculator1101, and based on the area information supplied from the areaspecifying unit 103, and then outputs the synthesized imagecorresponding to the background image or the foreground image.

FIG. 116 is a drawing showing a configuration of the synthesizer 1201.The same portions as those in the block diagram of FIG. 111 are givenwith the same reference numerals and the description thereof will beomitted.

The selector 1221 selects the estimated mixture ratio for the case thatsupposes the pixel belongs to the covered background area or theestimated mixture ratio for the case that supposes the pixel belongs tothe uncovered background area which are supplied from the mixture ratiocalculator 1101, based on the information supplied from the areaspecifying unit 103, and then supplies the selected estimated mixtureratio as the mixture ratio α to the background component generator 1021.

The background component generator 1021 shown in FIG. 116 generates abackground component image based on the mixture ratio α supplied fromthe selector 1221 and any background image, and then supplies thegenerated background component image to the mixed area imagesynthesizing portion 1022.

The mixed area image synthesizing portion 1022 shown in FIG. 116generates a mixed area synthesized image by synthesizing the backgroundcomponent image supplied from the background component generator 1021and the foreground component image, and then supplies the generatedmixed area synthesized image to the image synthesizing portion 1023.

The image synthesizing portion 1023 synthesizes the foreground componentimage, the mixed area synthesized image supplied from the mixed areaimage synthesizing portion 1022 and any background image, based on thearea information to generate and output the synthesized image.

Like the above, the synthesizer 1201 can synthesize the foregroundcomponent image and any background image.

Further, the mixture ratio α is explained as the ratio of the backgroundcomponent belonging to the pixel value but it may be a ratio of theforeground component belonging to the pixel value.

Further, it has been explained that the moving direction of the objectto be the foreground was from the left to right. But, of course, thedirection is not limited to that direction.

In above, the case when projecting the real space image having theinformation of the 3-dimensional space and time axis to the space-timehaving the information of the 2-dimensional space and time axis by usinga video camera was explained as an example, but the present invention isnot limited to this example and is applicable to the cases where thedistortion originated from the projection is corrected, the significantinformation is extracted or image is synthesized more naturally when thefirst information of the first dimension are projected to the secondinformation of the second dimension which is less than the firstinformation quantitatively.

Further, the sensor 76 a is not limited to the CCD and may be a solidstate image pickup element, for example, BBD (Bucket Brigade Device),CID (Charge Injection Device), CPD (Charge Priming Device), or CMOS(Complementary Mental Oxide Semiconductor) sensor. And the sensor is notlimited to the sensor where the detecting elements are arranged on thematrix, it may be a sensor where the detecting elements are adjacent ina row.

Each function of the separation server 11 explained above may beachieved by distributing the image processing to the various servers onthe network shown in FIG. 1. That is, the object extracting unit 101 andthe motion detector 102, may function as the motion detecting server 12the area specifying unit 103 may function as the area specifying server13, the mixture ratio calculator 104 may function as the mixture ratiocalculating server 14, the foreground/background separator 105 mayfunction as the foreground/background image separating server 15 and themotion blur adjusting unit 106 may function as the motion blur adjustingserver 16. Therefore, the block diagram of the separation server 11shown in FIG. 27 can be performed by hardware, software or network.Further, in the same manner, the synthesizing server 19 may function asthe synthesizer 1201 and the block diagram of the synthesizing processcan be performed by hardware, software or network.

Each processing of the object extracting unit 101, the motion detector102, the area specifying unit 103, the mixture ratio calculator 104, theforeground/background separation processor 105 and the motion bluradjusting unit 106 may be replaced with each processing of the motiondetecting server 12, the area specifying server 13, the mixture ratiocalculating server 14, the foreground/background image separating server15 and the motion blur adjusting server 16, and therefore theexplanation thereof is omitted.

Further, when performed by the hardware or software, the separationserver 11 may comprise of a portion included in the various servers, theclient computer 27 and the camera terminal units 28 which are connectedto the network in FIG. 1 as a separation processor. Thus, in thefollowing explanation, when the separation server 11 is explained as onedevice having the function for separating the input image into aforeground component image and a background component image, it isreferred to as the separation processor 11.

Next, referring the flow chart in FIG. 117, a process of separatingservice for the image input from the client computer 27 via the network1 of FIG. 1 by the separation server 11 will be explained.

In step S1001, the client computer 27 outputs the information forspecifying an image to the separation server 11. That is, a specificimage or an image ID specifying an image is output to the separationserver 11 as the information specifying an image desired by a user to beseparated.

In step S1011, the separation server 11 obtains a specified image. Thatis, the separation server 11 reads out and obtains the very imagetransmitted from the client computer 27, and reads out and obtains animage corresponding to an image ID when specifying information istransmitted from the client computer 27, via the network 1.

In step S1012, the charge processor 11 a of the separation server 11performs the charge processing via the network 1 by cooperating with theaccount charge server 24. Further, at the same time, in step S1021, theaccount charge server 24 performs the charge processing by cooperatingwith the separation server 11.

Here, referring to the flowchart of FIG. 118, the aforementioned chargeprocessing will be explained. Further, the real charge processing isperformed by the separation server 11 and the account charge server 24.However, since the information required for various processes is alsooutput from the client computer 27, here, the processing of the clientcomputer 27 will be explained together.

In step S1101, as shown in FIG. 119, the client computer 27 selects theservice and transmits a corresponding service fee to the separationserver 11 along with ID information for discerning the user (userprovided with image separating service) and the verification information(password, etc.) via the network 1. That is, at this time, when theinformation for specifying image is transmitted by means of theprocessing of step S1001 in FIG. 117, the processing in this step S1101is performed. Further, the service fee means the cost for the separatingservice.

In step S1111, as shown in FIG. 119, the charge processor 11 a of theseparation server 11 receives the ID information and the verificationinformation and transmits the service fee and its own ID to the accountcharge server 24.

In step S1121, as shown in FIG. 119, the account charge server 24inquires the verification information, the customer account ID and theservice fee from the financial transaction server 25 that is managed bya financial institution of the customer account on the basis of IDtransmitted from the separation server 11.

In step S1131, as shown in FIG. 119, the financial transaction server(for customer) 25 executes the verification processing on the basis ofthe customer account ID and the verification information and notifiesthe verified result and the information of authorization of use to theaccount charge server 24.

In step S1122, as shown in FIG. 119, the account charge server 24transmits the verified result and the information of authorization ofuse to the separation server 11. Further, in the following description,explanation will be given when the verified result is available.Further, when information about the non-availability of verified resultis received, the process ends.

In step S1112, as shown in FIG. 119, when the verified result and thefinancial institution is available, the separation server 11 providesthe client computer 27 with the service. In step S1102, the clientcomputer 27 receives the service provided. That is, at this time, instep S1112, the separation server 11 separates the specified image intoa foreground component image and a background component image to beoutput to the client computer 27 and in step S1102, the client computer27 receives the separated foreground component image and backgroundcomponent image.

In step S1113, the separation server 11 transmits the notification ofuse of the service to the account charge server 24. In step S1123, theaccount charge server 24 notifies the customer account ID, the servicefee and the provider account ID to the financial transaction server (forcustomer) 25.

In step S1132, the financial transaction server (for customer) 25transfers the service fee from the account of the customer account ID tothe financial transaction server (for provider) 26.

Here, the procedure returns to the explanation of the flowchart of FIG.117.

After the charge processing is performed by the separation server 11 andthe account charge server 24 in steps S1012, S1021, the separationserver 11 performs the image separation in step S1013. That is, the areaspecifying unit 103 of the separation server 11 perform the areaspecifying processing explained referring to the flowchart of FIG. 53,the mixture ratio calculator 104 performs the mixture ratio calculationprocessing explained by referring to the flowchart of FIG. 81, theforeground/background separator 105 performs the foreground/backgroundseparation processing explained referring to the flowchart of FIG. 96and the motion blur adjusting unit 106 performs the process of adjustingthe amount of motion blur explained referring to the flowchart of FIG.44, respectively, to separate the specified image. Further, since theamount-of-motion blur adjusting process, the area specifying process,the mixture ratio calculating process and the foreground/backgroundseparating process are similar to those described above, explanationthereof will be omitted.

In step S1014, the separation server 11 generates ID to the separatedforeground component image and the background component image and then,transmits it to the client computer 27. In step S1002, the clientcomputer 27 receives the foreground component image, the backgroundcomponent image and ID thereof transmitted from the separation server11, stores them in its own storage unit 48 (see FIG. 2) and prints outthem as needed. Further, the client computer 27 may make the separationserver 11 store in its own storage unit the foreground component imageand the background component image separated by the separation server 11or output them to the storage server 18 via the network 1 so as to storethem according to the instructions of a user.

In the above explanation, the separating process has been described whenthe service fee has been paid to the financial transaction servers 25,26 by the account charge server 24. However, the charge processing maybe performed, by storing points; such as pre-paid points, indicatingthat the user has previously paid the service fee to the provider of theseparating service in the storage unit 48 (see FIG. 2) and deducting thepoints at every time the user is provided with the separating service.

Here, referring to a flowchart of FIG. 120, the charge processing usingthe pre-paid points will be explained.

In step S1201, the client computer 27 specifies the service andtransmits the ID information and the verification information. That is,the client computer 27 performs the same processing as that in stepS1101 of FIG. 117.

In step S1211, the charge processor 11 a of the separation server 11receives the ID information and the verification information. In stepS1212, the charge processor 11 a subtracts the points corresponding tothe service fee for the separation processing from the pre-paid pointswhich is stored in storage unit 48 corresponding to the amount of moneypreviously paid by the user of the client computer 27 and stores it. Instep S1213, the separation server 11 provides the service. That is, atthis time, the separation server 11 performs the separation processingof the input image and transmits the foreground component image and thebackground component image separated to the client computer 27.

In step S1202, the client computer 27 receives the provided service.That is, at this time, the client computer 27, receives the foregroundcomponent image and the background component image transmitted from theseparation server 11.

Further, in the above explanation, a case that the separation server 11stores the pre-paid points in its own storage unit 48 (see FIG. 2) hasbeen described but for example, when a card in which the pre-paid pointsare stored, so called a pre-paid card is used, the same processing isperformed. In this case, it is required that the client computer 27 readout and transmits the pre-paid points stored in the pre-paid card instep S1201, the separation server 11 subtract the points correspondingto the service fee from the points received by the charge processing andthe points subtracted is transmitted to the client computer to updatethe pre-paid card.

Next, referring to the flowchart of FIG. 121, the processing of themotion detecting service for obtaining the motion vector and thepositional information of the image specified by the client computer 27will be explained.

In step S1301, the client computer 27 outputs the information specifyingan image to the motion detecting server 12. That is, as the informationspecifying the image desired by a user to be processed by the motiondetecting process, the specific image or the image ID specifying theimage is output to the motion detecting server 12.

In step S1311, the motion detecting server 12 obtains the specifiedimage. That is, the motion detecting server 12 reads out and obtains thevery image when an image is transmitted from the client computer 27 orthe image corresponding to the image ID when the specificationinformation is transmitted, via the network 1.

In steps S1312 and S1321, the charge processor 12 c of the motiondetecting server 12 and the account charge server 24 perform the chargeprocessing. Further, the charge processing is similar to that in theseparating service in FIG. 118 and FIG. 120 and thus, explanationthereof will be omitted.

In step S1313, the object extracting unit 12 a of the motion detectingserver 12 extracts each object from the specified image obtained, andthe motion detector 12 b detects the positional information and themotion vector, to transmit it to the client computer 27.

In step S1302, the client computer 27 receives and stores the positionalinformation and the motion vector of the object transmitted from themotion detecting server 12.

Further, the client computer 27 may store the positional information andthe motion vector detected by the motion detecting server 12 in its ownstorage unit or output them to the storage server 18 via the network 1so as to store them, in accordance with instructions of the user.

Next, referring to the flowchart of FIG. 122, the processing of the areaspecifying service specifying area from the information specifying theimage and object input from the client computer 27, which is performedby the area specifying server 13, will be explained.

In step S1401, the client computer 27 outputs the information specifyingthe image and the object to the area specifying server 13. That is, theinformation specifying the object is output to the area specifyingserver 13 along with the specific image or the image ID specifying theimage as the information specifying an image desired by a user to areaspecify.

In step S1411, the area specifying server 13 obtains the specifiedimage. That is, the area specifying server 13 reads out and obtains thevery image when the image is transmitted from the client computer 27 andthe image corresponding to the image ID when the image ID specifying animage is transmitted, via the network 1.

In steps S1412 and S1421, the charge processor 13 a of the areaspecifying server 13 and the account charge server 24 performs thecharge processing. Further, the charge processing is similar to that inthe separating service in FIG. 118 and FIG. 120 and thus, explanationthereof will be omitted.

In step S1413, the area specifying server 13 performs the areaspecifying processing on the basis of the information specifying theobject. Further, the area specifying processing is similar to theprocessing explained referring to the flowchart of FIG. 53 and thus,explanation thereof will be omitted.

In step S1414, the area specifying server 13 transmits the areainformation obtained through the processing in step S1413 to the clientcomputer 27.

In step S1402, the client computer 27 receives and stores the areainformation transmitted from the area specifying server 13.

Further, the client computer 27 may store the area information obtainedby the area specifying server 13 in its own storage unit or output it tothe storage server 18 via the network 1 so as to store it, in accordancewith instructions of the user.

Next, referring to a flowchart of FIG. 123, the processing of themixture ratio calculating service of calculating the mixture ratio fromthe information specifying the image and the object and the areainformation input from the client computer 27, which is performed by themixture ratio calculating server 14, will be explained.

In step S1501, the client computer 27 outputs, the informationspecifying the image and the object and the area information to themixture ratio calculating server 14. That is, as the informationspecifying an image of which mixture ratio is desired by a user tospecify, the specific image or the image ID specifying the image, theinformation specifying the object and the area information are output tothe mixture ratio calculating server 14.

In step S1511, the mixture ratio calculating server 14 obtains thespecified image. That is, the mixture ratio calculating server 14 readsout and obtains the very image when the image is transmitted from theclient computer 27 and the image corresponding to the image ID when theimage ID specifying an image is transmitted, via the network 1.

In steps S1512 and S1521, the charge processor 14 a of the mixture ratiocalculating server 14 and the account charge server 24 performs thecharge processing. Further, the charge processing is similar to that inthe separating service in FIG. 118 and FIG. 120 and thus, explanationthereof will be omitted.

In step S1513, the mixture ratio calculating server 14 performs themixture ratio calculation processing on the basis of the informationspecifying the object and the area information. Further, the mixtureratio calculation processing is similar to the processing explainedreferring to the flowchart of FIG. 81 and thus, explanation thereof willbe omitted.

In step S1514, the mixture ratio calculating server 14 transmits themixture ratio obtained through the processing in step S1513 to theclient computer 27.

In step S1502, the client computer 27 receives and stores the mixtureratio transmitted from the mixture ratio calculating server 14.

Further, the client computer 27 may store the mixture ratio obtained bythe mixture ratio calculating server 14 in its own storage unit oroutput it to the storage server 18 via the network 1 so as to store it,in accordance with instructions of the user.

Next, referring to the flowchart of FIG. 124, the processing of servicefor separating the foreground component image and the backgroundcomponent image from the information specifying the image and theobject, the area information and the mixture ratio input from the clientcomputer 27, which is performed by the foreground/background imageseparating server 15, will be explained.

In step S1601, the client computer 27 outputs, the informationspecifying the image and the object, the area information and theinformation of mixture ratio to the foreground/background imageseparating server 15. That is, as the information specifying an imagedesired by a user to separate into foreground and background, theconcrete image or the image ID specifying the image, the informationspecifying the object, the area information and the information ofmixture ratio are output to the foreground/background image separatingserver 15.

In step S1611, the foreground/background image separating server 15obtains the specified image. That is, the foreground/background imageseparating server 15 reads out and obtains the very image when the imageis transmitted from the client computer 27 and the image correspondingto the image ID when the image ID specifying an image is transmitted,via the network 1.

In steps S1612 and S1621, the charge processor 15 a of theforeground/background image separating server 15 and the account chargeserver 24 performs the charge processing. Further, the charge processingis similar to that in the separating service in FIG. 118 and FIG. 120and thus, explanation thereof will be omitted.

In step S1613, the foreground/background image separating server 15performs the foreground/background separating processing on the basis ofthe information specifying the object, the area information and themixture ratio. Further, the foreground/background separating processingis similar to the processing explained referring to the flowchart ofFIG. 96 and thus, explanation thereof will be omitted.

In step S1614, the foreground/background image separating server 15generates IDs to the foreground component image and the backgroundcomponent image obtained through the processing in step S1613 andtransmits them to the client computer 27.

In step S1602, the client computer 27 receives and stores the foregroundcomponent image and the background component image transmitted from theforeground/background image separating server 15.

Further, the client computer 27 may store the foreground component imageand the background component image transmitted from theforeground/background image separating server 15 in its own storage unitor output it to the storage server 18 via the network 1 so as to storeit, in accordance with instructions of the user.

Next, referring to the flowchart of FIG. 125, the processing of servicefor adjusting the motion blur of the specified image from theinformation specifying the image, the motion vector and the amount bywhich motion blur is adjusted input from the client computer 27, whichis performed by the motion blur adjusting server 16, will be explained.

In step S1701, the client computer 27 outputs the information specifyingthe image, the motion vector and the amount by which motion blur isadjusted to the motion blur adjusting server 16. That is, as theinformation specifying the image of which the motion blur is desired bya user to adjust, the specific image or the image ID specifying theimage, the information specifying the object, the motion vector andinformation of the amount by which motion blur is adjusted are output tothe motion blur adjusting server 16.

In step S1711, the motion blur adjusting server 16 obtains the specifiedimage. That is, the motion blur adjusting server 16 reads out andobtains the very image when the image is transmitted from the clientcomputer 27 and the image corresponding to the image ID when the imageID specifying an image is transmitted, via the network 1.

In steps S1712 and S1721, the charge processor 16 a of the motion bluradjusting server 16 and the account charge server 24 performs the chargeprocessing. Further, the charge processing is similar to that in theseparating service in FIG. 118 and FIG. 120 and thus, explanationthereof will be omitted.

In step S1713, the motion blur adjusting server 16 performs the motionblur adjusting processing on the basis of the motion vector and theamount by which motion blur is adjusted. Further, the motion bluradjusting processing is similar to the processing explained referring tothe flowchart of FIG. 104 and thus, explanation thereof will be omitted.

In step S1714, the motion blur adjusting server 16 generates ID for themotion blur adjusted image obtained through the processing in step S1713and transmits it to the client computer 27.

In step S1702, the client computer 27 receives and stores the motionblur adjusted image transmitted from the motion blur adjusting server16.

Further, the client computer 27 may store the motion blur adjusted imagetransmitted from the motion blur adjusting server 16 in its own storageunit or output it to the storage server 18 via the network 1 so as tostore it, in accordance with instructions of the user.

Next, referring to FIG. 126, detailed configuration of the encodingserver 17 will be explained. The separating portion 2002 of the encodingserver 17 separates the input image (including the image of which theimage ID specifying the image is input and which is read out from thestorage server 18 via the network 1) into the foreground component imageand the background component image and outputs it to the encoder 2001along with the mixture ratio, the motion vector and the positionalinformation. The separation processing portion 2002 has configurationsimilar to that of the separation server (separator) 11 explainedreferring to FIG. 27 and its processing of obtaining the mixture ratio,the motion vector and the positional information is similar to that ofthe separation server 11 and thus, explanation thereof will be omitted.

The encoder 2001 outputs the foreground component image and thebackground component image input from the separation processing portion2002 to the storage server 18 via the network 1 to make store them,converts them into the positional information stored on the network ofthe storage server 18, that is, the information such as URL and outputsit as the positional information of foreground component image and thepositional information of background component image. At that time, theencoder 2001 outputs the mixture ratio, the motion vector and thepositional information extracted by separating the foreground componentimage and the background component image.

When the foreground component image and the background component imageare converted into the positional information of foreground componentimage and the positional information of background component image,respectively, by the encoder 2001, the charge processor 17 a (see FIG.16 and FIG. 17) performs the charge processing with respect to theaccount charge server 24 via the network 1. Further, the chargeprocessing may be performed by the user to be provided with thesynthesizing service for generating the synthesized image, using thesynthesizing server 19 to be described later. Further, inversely, theuser using the encoding service may previously pay the service fee inadvance, so that paying for the service fee at the time of using thesynthesizing service, may be avoided by the user.

Next, referring to a flowchart of FIG. 127, the processing of theencoding service for encoding the image input from the client computer27, which is performed by the encoding server 17, will be explained.Further, in this explanation, a case that the user of the encodingservice pays the service fee will be described.

In step S1801, the client computer 27 outputs the information specifyingthe image to the encoding server 17. That is, as the informationspecifying the image desired by a user to encode, the specific image orthe image ID specifying the image and the information specifying theobject are output to the encoding server 17.

In step S1811, the encoding server 17 obtains the specified image. Thatis, the encoding server 17 reads out and obtains the very image when theimage is transmitted from the client computer 27 and the imagecorresponding to the image ID when the image ID specifying an image istransmitted, via the network 1.

In steps S1812 and S1821, the charge processor 17 a of the encodingserver 17 and the account charge server 24 perform the chargeprocessing. Further, the charge processing is similar to that in theseparating service in FIG. 118 and FIG. 120 and thus, explanationthereof will be omitted.

In step S1813, the separation portion 2002 of the encoding server 17performs the image separating processing. Further, the image separatingprocessing is similar to the processing explained referring to stepS1013 in the flowchart of FIG. 117 and thus, explanation thereof will beomitted.

In step S1814, the encoding server 17 outputs the foreground componentimage and the background component image obtained through the processingin step S1813 to the storage server 18 so as to store them. In stepS1813, the storage server 18 stores the foreground component image andthe background component image transmitted.

In step S1815, the encoding server 17 adds the motion vector and thepositional information to the positional information of foregroundcomponent image and the positional information of background componentimage generated through the encoding processing and transmits them tothe client computer 27.

In step S1802, the client computer 27 receives and stores the positionalinformation of foreground component image, the positional information ofbackground component image, the motion vector and the positionalinformation transmitted from the encoding server 17.

Further, in a case of separating and encoding the input image, theencoder 2001 may add only data corresponding to the difference to thecode (positional information of the image) of the image already encodedto output it when image similar to a pre-encoded image is encoded. Forexample, in a case of synthesizing the image shown in FIG. 128, when theencoding information of a first image including encoding information ofa foreground component image 1, a foreground component image 2 and amixture ratio 1 and the encoding information of a second image includingencoding information of a foreground component image 1, a foregroundcomponent image 3 and a mixture ratio 2 are synthesized, since theforeground component image 1 is also included in any image information,the foreground component image 1 of any of the images may be omittedwhen synthesizing and consequently, the compression rate can be improvedas much as the information of the foreground component image 1 to beomitted, compared with a case that both images are simply synthesized.

Consequently, when the first image and the second image shown in FIG.128 are stored, if the first image is first stored, only the encodinginformation of the mixture ratio 2 and the foreground component image 3that is a difference may be stored for the second image. For thisreason, when the encoding information of the same image is storedrepeatedly, the compression rate improves as the number of the storedimages increases.

Further, the mixture ratio, the motion vector and the positionalinformation to be encoded by the encoding server 17 may be theinformation specified by a user, as shown in FIG. 129. Further, as shownin FIG. 129, for the image to be encoded, the foreground component imageand the background component image corresponding to the image IDspecified by the user may be read out from the storage server 18 andencoded. In this case, the encoding server 17 may not be provided withthe separation portion 2002.

Further, although the image ID has been employed as the informationspecifying image in the present invention, the image positionalinformation may be employed instead.

Next, referring to a flowchart of FIG. 130, the processing of servicefor synthesizing the specified images A and B from the informationspecifying the images A and B input from the client computer 27, themotion vector, the mixture ratio, the positional information and theamount by which motion blur is adjusted, which is performed by thesynthesizing server 19, will be explained.

In step S1901, the client computer 27 outputs the information specifyingthe images A and B, the motion vector, the mixture ratio, the positionalinformation and the information of amount by which motion blur isadjusted to the synthesizing server 19. That is, as the informationspecifying the images A and B desired by a user to synthesize, thespecific image or the image A ID and B ID (may be the aforementionedpositional information of encoded images) specifying the images A and B,the motion vector, the mixture ratio, the positional information and theinformation of amount by which motion blur is adjusted are output to thesynthesizing server 19.

In step S1911, the synthesizing server 16 obtains the specified images.That is, the synthesizing server 19 reads out and obtains the very imagewhen the images are transmitted from the client computer 27 and theimages corresponding to the image IDs when the image IDs specifyingimages are transmitted, via the network 1.

In steps S1912 and S1921, the charge processor 19 a of the synthesizingserver 19 and the account charge server 24 perform the chargeprocessing. Further, the charge processing is similar to that in theseparating service in FIG. 118 and FIG. 120 and thus, explanationthereof will be omitted. Further, the charge processing may be omittedin a case that the user accepting the encoding service has been chargedby using the encoding server 16. Inversely, in place of the useraccepting the encoding service, the user accepting the synthesizingservice may be charged.

In step S1913, the synthesizing server 19 performs the processing ofsynthesizing the images A and B on the basis of the motion vector, themixture ratio, the positional information and the information of amountby which motion blur is adjusted.

In step S1914, the synthesizing server 19 generates ID for thesynthesized image (A+B) obtained through the processing in step S1913and transmits it to the client computer 27.

In step S1902, the client computer 27 receives and stores thesynthesized image (A+B) transmitted from the synthesizing server 19.

Further, the client computer 27 may store the synthesized image (A+B)transmitted from the synthesizing server 19 in its own storage unit oroutput it to the storage server 18 via the network 1 so as to store it,in accordance with instructions of the user.

As described above, although the synthesizing server 20 is made to beable to synthesize a plurality of images it may generate the encryptedimage by adding the motion blur of the synthesized image using theamount by which motion blur is adjusted as a key. FIG. 131 shows aconfiguration of the motion blur adder 2021 for encryption, provided tomake the synthesizing server 20 generate the encrypted image.

The input information processor 2031 of the motion blur adder forencryption 2021 outputs the input signal to be encrypted to the imagingportion 2032, outputs information of an encrypting key to the motionblur adder 2033 and outputs the image selection information forselecting image (background component image) desired to synthesize byusing the signal to be encrypted as the foreground component image tothe synthesizing server 20, respectively.

When the signal to be encrypted input from the input informationprocessor 2031 is not image signal, the imaging portion 2032 convertsthe signal into the image signal to output it to the motion blur adder2033. That is, since the encrypted signal is supposed to be imagesignal, the imaging portion 2032 images the signal that is not imagesignal in order to correspond with such processing.

The motion blur adder 2033 generates the amount by which motion blur isadjusted on the basis of the information such as speed or directioninput from the input information processor 2031 and adds the motion blurto the image signal input from the imaging portion 2032 to output it tothe synthesizing server 20. The synthesizing server 20 obtains thebackground component image on the basis of the image selectioninformation input from the input information processor 2031 andsynthesizes the image input from the motion blur adder 2033 as theforeground component image with the obtained background component imageto generate and display the synthesized image. At this time, the imageselection information specifying the background component image may bethe background component image itself and may be the positionalinformation of background component image or ID of the backgroundcomponent image.

Next, referring to FIG. 132, the motion blur eliminating portion forencryption 2041 for decrypting the synthesized image encrypted by themotion blur adder for encryption 2021 provided in the synthesizingserver 20 to covert it into the original signal will be explained.Further, the motion blur adder for encryption 2021 and the motion blureliminating portion for encryption 2041 shown in FIG. 131 and FIG. 132may be thought, for example, to be a functional block diagram ofsoftware built-in to the client computer 27 and to be a block diagram ofhardware. Further, the motion blur adder for encryption 2021 and themotion blur eliminating portion for encryption 2041 may be constructedas an exclusive server on the network 1.

The separation server 11 separates the encrypted synthesized image intothe foreground component image and the background component image andoutputs the foreground component image to which motion blur is added tothe input information processor 2051.

When the encrypted foreground component image input from the separationserver 11 and the information on speed and direction as a key fordecrypting the encrypted foreground component image are input, the inputinformation processor 2051 output them to the motion blur eliminatingportion 2052. When the image is displayed in 2-dimensions of x directionand y direction, the speed and the direction of the key are set thereto,respectively.

The motion blur eliminating portion 2052 generates the amount of motionblur on the basis of the information of speed and direction input fromthe input information processor 2051, performs the motion blur addingprocessing inverse to the motion blur adding processing performed by themotion blur adder for encryption 2021 on the encrypted foregroundcomponent image, and decrypts the encrypted foreground component imageto output it to the signal converter 2053. When the encrypted signal tobe encrypted is not image signal, the signal converter 2053 coverts theimage signal input from the motion blur eliminating portion 2052 intothe original signal to output it.

That is, the motion blur adder 2033 (see FIG. 131) and the motion blureliminating portion 2052 performs substantially the processing similarto the motion blur adder 806 in FIG. 97 and performs the motion bluradding processing inverse to each other using the amount by which motionblur is adjusted as the encrypting key. However, they are different inthat gain-up processing is further performed on the motion blur addingprocessing first performed in the motion blur adding processes in xdirection or y direction to be described below.

Here, principles in which the image signal is encrypted by adding themotion blur will be explained. For example, as shown in FIG. 133, whenthe subject is moving in the arrow direction, picture thereof is takenby the sensor 76 a having CCD, etc., the mixed area (covered backgroundarea and uncovered background area) is generated ahead and behind in themoving direction as the motion blur of the picked up image (see FIG. 31Afor details). FIG. 134 shows an example illustrating this phenomenon, inwhich for a case that the subject indicated by A in FIG. 134 is pickedup by the sensor 76 a, when the subject is moving in left or rightdirection in the drawing, area of motion blur is extended and the pickedup color is spread according to the speed thereof. That is, when thesubject is moving at speed of v in left or right direction, the imageindicated by B in FIG. 134 is supposed to be picked up. At this time, ifareas picked up when the subject is stationary are the areas a0 to a0′and areas where the subject indicated by B in FIG. 134 are a1 to a1′,the colors of the areas of a0 to a0′ at the original position getlighter and are spread in the areas a1 to a0 and the areas a0′ to a1′where the motion blur is generated. Similarly, if the subject is movingat a speed of 2v (2 times v), as indicated by C in FIG. 134, it can beshown that the colors are spread in the areas a2 to a0 and the areas a0′to a2′ where the motion blur is generated. Further, if the subject ismoving at a speed of 3v, as indicated by D in FIG. 134, the colors arespread in the areas a3 to a0 and the areas a0′ to a3′ where the motionblur is generated and if the subject is moving at a speed of 4v, asindicated by E in FIG. 134, the colors are spread in the areas a4 to a0and the areas a0′ to a4′ where the motion blur is generated and as awhole, the colors get lighter. That is, since the respective pixelvalues output from the sensor 76 a result from integrating with respectto the shutter time, the portions of the picked up object havingmovement is spread spatially as the speed of the moving subjectincreases to cause the colors in that region to lighten, in comparisonthe whole integrated pixel value are spatially spread with a slightvariation. Therefore, as the area where the colors get lighter isspread, and the area of motion blur expands, and resolution of thesubject decreases. Here, color indicates resolution. As color becomesthicker, the resolution increases and as color becomes lighter, theresolution decreases.

The encryption using the motion blur adjustment employs this nature andis performed by generating motion blur in the image in the 2-dimensionaldirection, which cannot be generated in the real world. That is, asshown in FIG. 135, the image obtained by taking picture of a subjecthaving a black circle shape in a stationary state is displayed in theleftmost column and uppermost row in the matrix. In this state, forexample, if the motion blur in moving vertically is added, the blackcircle shaped subject is an image in which the motion blur is generatedin the vertical direction, as shown in the center column and uppermostend. Further, if the motion blur is generated horizontally, the image inwhich the motion blur is generated in the vertical and horizontaldirections of the subject is obtained as shown in the center column andcenter row.

In this state, furthermore, if movement (speed) in the horizontaldirection increases to add the motion blur, as shown in the centercolumn and lowermost row, the image in which the motion blur area isspread in the horizontal direction is further obtained. If the motionblur is generated in the vertical direction in this image, as shown inthe rightmost column and lowermost row, the motion blur area of theblack circle shaped subject is spread and as a whole, and the colorsbecome lighter. As a result, since resolution of the subject isdecreased, it is possible to encrypt the image itself.

Next, referring to the flowchart of FIG. 136, the encrypting processingusing the amount by which motion blur is adjusted by the motion bluradder for encryption 2021 will be explained. Further, in the followingexplanation, as shown in FIG. 137, an example of encrypting the imageobtained by taking picture of the subject comprised 5×5 pixels will beexplained. Here, in FIG. 137, the respective pixels of 5×5 pixels areindicated by the pixel values a to y, the vertical direction isindicated by y, the horizontal direction is indicated by x and thetime-axis is indicated by time t.

In step S2001, the input information processor 2031 determines whetherthe signal to be encrypted is input or not, the processing is repeateduntil it is input and if it is determined that it is input, theprocedure proceeds to step S2002.

In step S2002, the input information processor 2031 outputs the inputsignal to be encrypted to the imaging portion 2032. In step S2003, theimaging portion 2032 determines whether the input signal to be encryptedis image signal or not. For example, if it is determined that the signalto be encrypted is not image signal, the imaging portion 2032 convertsthe signal to be encrypted into image signal to output it to the motionblur adder 2033 in step S2004. In step S2003, if it is determined thatthe information to be encrypted is image signal, the imaging portion2032 outputs the input signal to be encrypted to the motion blur adder2033.

In step S2005, the input signal processor 2031 determines whether thespeed and direction information as a key is input or not and repeats theprocessing until it is input, and if the speed and direction as a key isinput, the procedure proceeds to step S2006.

In step S2006, the motion blur adder 2033 encrypts the input imagesignal in the x direction (adds the motion blur).

Here, referring to FIGS. 137 to 149, a method for generating specificpixel values in encrypting the subject by adjusting the motion bluradjustment will be explained.

Here, as shown in FIG. 137, a method for encrypting the pixels a to e inthe lowermost row by generating the motion blur in the x direction willbe explained. At this time, if the amount of movement v indicating thespeed as a key is 5 (the virtual division number is 5), the pixels inthe lowermost row shown in FIG. 138 is shown similar to those in FIG.139. That is, since the respective pixel values are divided by 5 in thetime direction, the relationships of a/5=a0=a1=a2=a3=a4,b/5=b0=b1=b2=b3=b4, c/5=c0=c1=c2=c3=c4, d/5=d0=d1=d2=d3=d4 ande/5=e0=e1=e2=e3=e4 are satisfied. Here, the pixel values in theuppermost row in FIG. 139 are the pixel values for a time just ahead.

If movement in the x direction (in here, the right side in the drawing)is added to the subject, the arrangement of the pixel values shifts at apredetermined time interval and as a result, an arrangement shown inFIG. 140 is obtained. That is, the arrangement in which the pixel valuesare shifted according to movement of the subject is obtained such thatat the time that movement starts, the pixel values a0 to e0 are at theoriginal positions, the pixel values al to e1 shift by one pixel in theright direction at the next time interval, the pixel values a2 to e2shift by one pixel in the right direction at the next time interval, thepixel values a3 to e3 shift by one pixel in the right direction at thenext time interval and the pixel values a4 to e4 shift by one pixel inthe right direction at the next time interval.

Further, the respective pixel values on the x y plane are obtained bysumming the pixel values shown in FIG. 140 in the time direction.However, for example, since the leftmost column or the rightmost columncomprises only the pixel value a0 or e4, there is a problem that thevalue of a pixel becomes very small. Then, after the same processing isperformed in the y direction, the gain-up processing is furtherperformed in order not to be very small pixel value. An example of thegain-up processing performed is shown in FIG. 141.

Here, a0*=5×a0, b0*=(5/2)×b0, a0*=(5/2)×a1, c0*=(5/3)×c0, b1*=(5/3)×b1,a2*=(5/3)×a2, d0*=(5/4)×d0, c1*=(5/4)×c1, b2*=(5/4)×b2, a3*=(5/4)×a3,e1*=(5/4)×e1, d2*=(5/4)×d2, c3*=(5/4)×c3, b4*=(5/4)×b4, e2*=(5/3)×e2,d3*=(5/3)×d3, c4*=(5/3)×c4, e3*=(5/2)×e3, d4*=(5/2)×d4 and e4*=5×e4.That is, the weight of each pixel adjusts the gain to be the pixel valueby 1 pixel. As a result, if the pixels a to e shown in FIG. 138 areencrypted on a condition that the amount of movement v in the xdirection is 5 (if the motion blur is added), they are converted intothe pixels ax to dx′ (are encrypted) as shown in FIG. 142 and the numberof pixels of the subject in the horizontal direction is increased from 5to 9. Here, the pixel number is ax=ax*, bx=(b0*)+(a1*),cx=(c0*)+(b1*)+(a2*), dx=(d0*)+(c1*)+(b2*)+(a3*),ex=(e0)+(d1)+(c2)+(b3)+(a4), ax′=(e1*)+(d2*)+(c3*)+(b4*),bx′=(e2*)+(d3*)+(c4*), cx′=(e3*)+(d4*) and ex=ex*.

If the aforementioned processing encrypts all the rows in the ydirection comprised of the 5×5 pixels shown in FIG. 137 in the xdirection, the pixel values shown in FIG. 143 are obtained. That is, thepixels ax to yx and the pixels ax′ to dx′, fx′ to ix′, kx′ to nx′, px′to sx′ and ux′ to xx′ are obtained, and spread in the x direction isgenerated due to generation of the motion blur in the x direction. Thus,the pixel values are obtained at every 9 pixels.

Here, returning to the flowchart of FIG. 136, the explanation will becontinued.

In step S2007, the motion blur adder 2033 encrypts the image signalencrypted in the x direction with respect to the y direction.

Here, as shown in FIG. 144, a method for encrypting the pixels ax, fx,kx, px, ux in the rightmost column in FIG. 143 by generating the motionblur in the y direction will be explained. At that time, if the amountof movement v indicating the speed as a key is 5 (the virtual divisionnumber is 5), the pixels in the rightmost column in FIG. 143 isdisplayed as in FIG. 144. That is, since the respective pixel values aredivided into 5 in the time direction, the relationships ofax/5=ax0=ax1=ax2=ax3=ax4, fx/5=fx0=fx1=fx2=fx3=fx4,kx/5=kx0=kx1=kx2=kx3=kx4, px/5=px0=px1=px2=px3=px4 andux/5=ux0=ux1=ux2=ux3=ux4 are satisfied. Here, the pixel in the uppermostrow in FIG. 145 are the pixel values for a time just ahead.

If movement in the y direction is added to the subject, the arrangementof the pixel values shifts at a predetermined time interval and as aresult, an arrangement shown in FIG. 146 is obtained. That is, thearrangement in which the pixel values are shifted according to movementof the subject is obtained such that at the time that movement starts,the pixel values ax0, fx0, kx0, px0 and ux0 are at the originalpositions, the pixel values ax1, fx1, kx1, px1 and ux1 shift by onepixel in the right direction at the next time interval, the pixel valuesax2, fx2, kx2, px2 and ux2 shift by one pixel in the right direction atthe next time interval, the pixel values ax3, fx3, kx3, px3 and ux3shift by one pixel in the right direction at the next time interval andthe pixel values ax4, fx4, kx4, px4 and ux4 shift by one pixel in theright direction at the next time interval.

Here, returning to the flowchart of FIG. 136, the explanation will becontinued.

In step S2008, the synthesizing server 19 synthesizes the synthesizedbackground component image with the encrypted image (foregroundcomponent image). For example, if the background component image (imagehaving pixels of one step in the x direction) comprised of the pixelvalues B0 to B9 arranged in the y direction in FIG. 147 is synthesized,the pixel values are values to which the pixel values shown in FIG. 148are added. That is, the pixel values of the image (to which the motionblur is added) are encrypted in the x and y directions as the pixelvalues of the foreground component image, and the pixel values of theimage to be synthesized as the pixel values of the background componentimage are synthesized. As a result, the pixel values A, F, K, P, U, Ay′,Fy′, Ky′ and Py′ shown in FIG. 149 are obtained and the respective pixelvalues are the pixel value A=ax0+B0×4/5, the pixel valueF=fx0+ax0+B1×3/5, the pixel value K=kx0+fx1+ax2+B2×2/5, the pixel valueP=px0+kx1+fx2+ax3+B3×2/5, the pixel value U=ux0+px1+kx2+fx3+ax4, thepixel value Ay′=B5×1/5+ux1+px2+kx3+fx4, the pixel valueFy′=B6×2/5+ux2+px3+kx4, the pixel value Ky′=B7×3/5+ux3+px4 and the pixelvalue Py′=B8×4/5+ux4.

By performing all these processes in all the y direction, thesynthesized image of which the background component image is theencrypted foreground component image shown in FIG. 150 is generated.That is, the input image of 5×5 pixels is converted into the image of9×9 pixels (the pixels A to Y, the pixels Ax to Dx, the pixels Fx to Ix,the pixels Kx to Nx, the pixels Px to Sx, the pixels Ux to Xx, thepixels Ay′ to Ty′, the pixels Ax′ to Dx′, the pixels Fx′ to Ix′, thepixels Kx′ to Nx′ and the pixels Px′ to Sx′).

Further, since is the motion blur adding processing in the motion blureliminating portion for encryption 2041, in the reverse of theprocessing by the motion blur adder for encryption 2021, explanation ofthe decrypting processing will be omitted.

Further, in the processing of step S2006, when encryption in the xdirection is performed; the gain-up processing is performed and then,encryption in the y direction is performed. Therefore, it is necessarythat after decryption in the y direction is performed, the gain islowered, and then the decryption in the x direction is performed.Further, the order of the encryptions in the x direction and the ydirection may be changed. However, since the gain-up processing isperformed in the direction where the encryption is formerly performed,it is necessary that the order of decryptions should correspond to theorder of encryptions.

Next, referring to the flowchart of FIG. 151, the encrypting service bythe synthesizing server 19 comprising the motion blur adder forencryption 2021 shown in FIG. 131 will be explained. Further, this is aprocessing that the client computer 27-1 connected to the network 1transmits the signal to be encrypted to the synthesizing server 19,which encrypts it and transmits it to the client computer 27-2. Further,the client computer 27 may be provided with a hardware having a functionof the image separation processing of the separation server 11 havingthe motion blur eliminating portion for encryption 2041 or installedwith software having such function.

In step S2101, the client computer 27-1 transmits information desired toencrypt (signal to be encrypted), information of speed and direction asthe encryption key and image selection information (informationselecting the background component image) to the synthesizing server 19.

In step S2111, the motion blur adder for encryption 2021 of thesynthesizing server 19 encrypts the information desired to encrypt(signal to be encrypted) on the basis of the encryption key input fromthe client computer 27-1 and performs the encrypting processing ofsynthesizing the selected background component image. Further, since theencrypting processing was explained referring to the flowchart of FIG.136, explanation thereof will be omitted.

In step S2112, the synthesizing server 19 transmits a synthesized imageencrypted by adding the motion blur to the client computer 27-1.

In step S2102, the client computer 27-1 displays the synthesized imagereceived from the synthesizing server 19 to determine whether the imageis desired by the user or not and if it is determined that the image isthe desired image, it is notified to the synthesizing server 19 that theimage is the desired image in step S2103. In step S2113, thesynthesizing server 19 determines whether the image is the desired imageor not and for example, in this case, since it is notified to thesynthesizing server 19 that the image is the desired image in stepS2103, the procedure proceeds to step S2114.

In step S2114, the charge processor 19 a of the synthesizing server 19and the account charge server 24 perform the charge processing. Further,the charge processing is similar to that of the separating service inFIGS. 118 and 120 and thus, explanation thereof will be omitted.

In step S2115, the synthesizing server 19 transmits the synthesizedimage encrypted to the client computer 27-1. In step S2104, the clientcomputer 27-1 receives the encrypted synthesized image and transmits itto the client computer 27-2.

In step S2141, the client computer 27-2 receives the encryptedsynthesized image. In step S2142, the client computer 27-2 determineswhether the key is input or not and repeats this processing until theencryption key is input. If the information of speed and direction asthe encryption key is input in step S2142, the motion blur eliminatingportion 2041 performs the motion blur processing on the basis of thespeed and the direction input in step 2143. In step S2144, the image inwhich the motion blur is eliminated is displayed.

If it is determined that the image was not the desired image in stepS2102, it is notified to the synthesizing server 19 in step S2015 thatthe image is not the desired image and the procedure returns to stepS2101. Further, since it is determined by this processing that the imageis not the desired image, the processing of the synthesizing server 19returns to step S2111.

That is, by this processing, when the key of speed and directionspecified by a user is correctly input to the client computer 27-1, animage that the encrypted image has been correctly decrypted isdisplayed. Further, through the same system as the aforementionedencrypting service, the decrypting service can be provided.

Next, referring to FIG. 152, the correcting server 20 will be explained.

The separation processor 11 of the correcting server 20 separates theinput image (which is may be the image ID and when the image isspecified by the image ID, the corresponding image retrieved from thenetwork 1) into the foreground component image and the backgroundcomponent image to output the foreground component image to the motionblur adjusting portion 2101 and to output the background component imageto the synthesizer. The motion blur adjusting portion 2101 adjusts themotion blur of the input foreground component image by the specifiedamount of motion blur (adjusting amount of correction) and outputs it tothe synthesizer 2101. The synthesizer 2101 synthesizes the foregroundcomponent image in which motion blur is adjusted and the inputbackground component image and output it as the corrected image.

For example, the image shown in FIG. 153A is supposed to be input to thecorrecting server 20. That is, as shown in right portion of FIG. 153A,when the foreground is moved over the background in the arrow direction,the motion blur is generated in the moving direction of the foregroundand the reverse direction thereof. This portion of motion blur is themixed area, the mixed area occurring in the leading portion in themoving direction is the CB (covered background) and the mixed areaoccurring in the trailing portion in the moving direction is UB(uncovered background) as shown in the left portion of FIG. 153A.Further, since the time axis t is set in the vertical direction in theleft portion of FIG. 153A, the relation between the state that the pixelvalues of pixels are stored and the time lapse is shown with movement.The separation processor 11 separates this input image into theforeground and the background as shown in FIG. 153B. At that time, themixed area of the input image is extracted simultaneously.

The motion blur adjusting portion 2101 adjusts the motion blur of theforeground component image shown in FIG. 153B to generate, for example,the foreground component image such as that in FIG. 153C. That is, inthis case, the motion blur is set to be small (CB and UB is set to besmall). Further, the amount of motion blur to be adjusted for adjustingthe motion blur may be input several times by the repeated operations ofthe user or may be set to a predetermined value by the motion bluradjusting portion 2101.

The synthesizer 2102 synthesizes the foreground component image adjustedas shown in FIG. 153C and the input background component image andsynthesizes and outputs the foreground component image in which themotion blur is adjusted with the background component image as shown inFIG. 153D.

Further, when the background component image is to be changed into otherbackground component image different from that of the input image, theseparated background component image is not input to the synthesizer2102 and the background component image desired to be changed is inputto the synthesizer 2102. Further, the correcting server 20 may beconstructed by replacing the separation processor 11, the motion bluradjusting portion 2101 and the synthesizer 2102 with the separationserver 11, the motion blur adjusting server 16 and the synthesizingserver 19 on the network 1.

Next, referring to a flowchart of FIG. 154, the processing of thecorrecting service for correcting the image input from the clientcomputer 27, which is performed by the correcting server 20, will beexplained.

In step S2201, the client computer 27 outputs the information specifyingthe image to the correcting server 20. That is, as the informationspecifying the image desired by a user to correct, the specific image orthe image ID specifying the image are output to the correcting server20.

In step S2211, the correcting server 20 obtains the specified imagedesired to correct and the background component images, and theseparation processor 11 separates the image desired to correct into aforeground component image and a background component image. That is,the correcting server 20 reads out and obtains the very image when theimage is transmitted from the client computer 27 and the imagecorresponding to the image ID when the image ID specifying an image istransmitter via the network 1. Further, the separation processor 11separates the obtained image into a foreground component image and thebackground component image.

In steps S2212 and S2221, the charge processor 20 a of the correctingserver 20 and the account charge server 24 perform the chargeprocessing. Further, the charge processing is similar to that in theseparating service in FIGS. 118 and 120 and thus, explanation thereofwill be omitted.

In step S2213, the motion blur adjusting portion. 2101 of the correctingserver 20 performs the process of motion blur adjustment of foregroundcomponent image. Further, the processing of adjusting motion-blur issimilar to the processing explained referring to the flowchart of FIG.104 and thus, explanation thereof will be omitted.

In step S2214, the synthesizer 2102 synthesizes the foreground componentimage in which motion blur is adjusted and the specified backgroundcomponent image. In step S2215, the correcting server 20 transmits thesynthesized image through the processing in step S2214, that is, thecorrected image to the client computer 27.

In step S2202, the client computer 27 stores the corrected imagetransmitted from the correcting server 20.

Further, the client computer 27 may store the image corrected by thecorrecting server 20 in its own storage unit or output the correctedimage to the storage server 18 so as to store it via the network 1 inaccordance with instructions of a user.

Next, referring to a flowchart of FIG. 155, the processing of the imagepurchasing service for purchasing the image specified by the clientcomputer 27, which is performed by the purchasing server 21, will beexplained.

In step S2301, the client computer 27 outputs the information specifyingthe image desired to purchase to the purchasing server 21. That is, asthe information specifying the image desired by a user to purchase, theimage ID specifying the image is output to the purchasing server 21.

In step S2311, the purchasing server 21 obtains the image desired topurchase. That is, the purchasing server 21 reads out and obtains theimage corresponding to the image ID transmitted from the client computer27 via the network 1.

In steps S2312 and S2321, the charge processor 21 a of the purchasingserver 21 and the account charge server 24 perform the chargeprocessing. Further, the charge processing is similar to that in theseparating service in FIGS. 118 and 120 and thus, explanation thereofwill be omitted.

In step S2313, the purchasing server 21 transmits the image obtainedthrough the processing in step S2311 to the client computer 27.

In step S2302, the client computer 27 receives and stores the imagetransmitted from the purchasing server 21.

Also, the client computer 27 may store the image purchased by thepurchasing server 21 in its own storage unit or output it to the storageserver 18 so as to store it via the network 1 in accordance withinstructions of a user. Further, the client computer 27 may transmit theimage to another client computer 27, for example, to present it.Furthermore, another user may purchase the foreground component image,the background component image, the synthesized image, the correctedimage or the like each of which is separated by the separationprocessing service, the synthesizing service or the correcting service.

Next, referring to a flowchart of FIG. 156, the processing of the imageselling service for selling the image specified by the client computer27, which is performed by the selling server 22, will be explained.

In step S2401, the client computer 27 outputs the information specifyingthe image desired to sell to the purchasing server 21. That is, theimage desired by user to sell is output to the selling server 22.

In step S2411, the selling server 22 obtains the image desired to sell.That is, the selling server 22 obtains the image transmitted from theclient computer 27.

In steps S2422, the selling server 22 sets a proper price on the imagedesired to sell. The price may be set, for example, by user in advance,or through an auction on the network 1. Further, if an image is aperson, the price may be set according to whether the person is awell-known figure or not when the image is selected.

In step S2413 and S2431, the charge processor 22 a of the selling server22 and the account charge server 24 perform the charge processing.

Here, referring to the flowchart of FIG. 157, the processing of thecharge will be explained. Further, the real processing of the charge isperformed by the selling server 22 and the account charge server 24.But, the information needed for various processing is also output fromthe client computer 27 and therefore the processing of the clientcomputer 27 will also be explained here.

In step S2501, an ID for discerning a user (who user who is selling animage) is transmitted to the selling server 22 via the network 1.

In step S2511, the selling server 22 transmits a cost and an ID fordiscerning the selling server 22 to the account charge server 24 on thebasis of the ID information transmitted from the client computer 27.

In step S2521, the account charge server 24 requests the financialtransaction server 26 having provider account to pay the cost to thefinancial transaction server 25 having customer account on the basis ofthe transmitted ID for discerning the selling server 22.

In step S2531, the financial transaction server for provider 26transfers the cost corresponding to the amount of sale from the provideraccount to the financial transaction server for customer 25 havingaccount of the customer.

Here, the flowchart of FIG. 156 will be explained again.

In step S2424, the selling server 22 notifies the end of sale to theclient computer 27. In step S2402, the client computer 27 receives thenotification of the end of sale.

Also, the selling server 22 may make the selling server 21 store theimage sold by the user in its own storage unit or output the image tothe storage server 18 so as to store it. Further, when the price is setthrough an auction system as mentioned above, the selling server 22 maymake the image sold by the user to the client computer of a successfulbidder 27.

Next, referring to the flowchart of FIG. 158, the retrieving server 23will be explained.

The retrieving server 23 retrieves the image which has been picked up onthe camera terminal unit 1 connected to the network 1 on the basis ofthe retrieving conditions input from the client computer 27 and outputthe request image. The retrieving condition is time, season, weather,area, position, subject or the like.

The control unit 2161 of the retrieving server 23 controls the wholeoperation of the retrieving server 23. In database 2162, position datawhich is obtained by GPS 76 b provided with a camera terminal unit 28corresponding to the camera ID of each camera terminal unit 28 which theretrieving server 23 recognizes, weather data, subject data or the like2062 b are stored as database. The contents of the database 2162 areobtained and updated from the camera terminal unit 28 via the network 1by controlling the communication unit 2165 at a predetermined timeinterval through the control unit 2161.

The storage unit 2163 stores the image obtained from the camera terminalunit 28 on the network 1 from the communication unit 2165 or storesinformation needed for the processing of various images.

The requested information generator 2164 arranges the retrievingconditions input from the client computer 27 on the network 1, and thengenerates the condition for retrieving the database 2162. That is, forexample, when season is put as the retrieving condition, the season maybe specified by the position data of each camera terminal unit 28 andthe time information calculated by the time calculator 2166. So, therequested information generator 2164, for example, in the case when‘spring’, is input as the retrieving condition, generates the positiondata of the latitude and longitude of the earth in which it is spring atthe present time. The control unit 2161 controls the communication unit2165, and then obtains the image corresponding to the ‘spring’ byreading out the pick up images of the camera terminal unit 28 of thecamera ID corresponding to the position data from the network 1.

The separation processor 2167 obtains the image desired to retrieve,belonging to the image which has been read out through the separationprocessing. Further, the separation processor 2167 has same functions asthose of the separation processor 11.

Next, referring to a flowchart of FIG. 159, the processing of theretrieving service for retrieving the image based on the retrievingconditions input from the client computer 27, which is performed by theretrieving server 23, will be explained.

In step S2601, the client computer 27 outputs the retrieving conditionsto the retrieving server 23. In step S2611, the retrieving server 23receives the retrieving conditions from the communication unit 2165.

In steps S2612 and S2631, the charge processor 23 a of the retrievingserver 23 and the account charge server 24 perform the chargeprocessing. Further, the charge processing is similar to that in theseparating service in FIGS. 118 and 120 and thus, explanation thereofwill be omitted. Further, the charge processing in steps S2612 and S2631is the change processing relating to the cost for performing theretrieval process.

In step S2613, the retrieving server 23 retrieves the imagecorresponding to the retrieving conditions and calls out thecorresponding image in step S2614. In step S2641, the camera terminalunit 28 transmits the image shown to the retrieving server 23.

That is, for example, supposing that the client computer 27-1 to 27-5,the retrieving server 23 and the camera terminal unit 28-1 to 28-5 wereconnected to the network 1 as shown in FIG. 160. At this time if theclient computer 27-2 transmits ‘person’, ‘car’ and ‘building’ as theretrieving conditions by the operation of a user to the processing ofstep 2611, the retrieving server 23 retrieves the objects of ‘person’,‘car’ and ‘building’ as the retrieving conditions in the database 2162in step 2613. That is, in case of FIG. 160, the retrieving server 23retrieves a car 2172 being picked up by the camera terminal unit 28-1whose ID is 1, a person 2183 being picked up by the camera terminal unit28-2 whose ID is 2 and a building being picked up by the camera terminalunit 28-5 whose ID is 5, and then obtains the images from each cameraterminal unit 28 in step S2614.

In step S2515, the retrieving server 23 determines whether the calledout image is separated or not, that is whether the image (object) whichdoes not satisfy the desired conditions is included or not.

In the case of FIG. 160, since a cloud 2172 which does not satisfy theretrieving conditions is included in the image transmitted from thecamera terminal unit 28-1 and a house which does not satisfy the desiredconditions is included in the image transmitted from the camera terminalunit 28-2, these images are not separated images corresponding to theretrieving conditions, and the procedure proceeds to the step S2616.

In step S2616, the separation processor 2167 performs the separationprocessing. Further, the separation processor is similar to theprocessing of the flowchart in FIG. 117 and thus, explanation thereofwill be omitted.

In step S2617, the called out images is synthesized and transmitted tothe client computer 27. In step S2602, the client computer 27 obtainsthe image transmitted from the retrieving server 23. In step S2603, theclient computer 27 determines whether the received image is the desiredimage or not. As shown in FIG. 160, in the case of the image shown indisplay 27 a-1, ‘person’, ‘car’ and ‘building’ as the retrievingconditions are included in the image as person 2182, car 2172 andbuilding 2211 because these images are the desired image, it is notifiedto the retrieving server 23 that the image is the desired image in stepS2604.

In step S2618, the retrieving server 23 determines whether it is thedesired image or not based on the notification transmitted from theclient computer 27. In the case of the FIG. 160, because it is desiredimage, the procedure proceeds to the step S2619.

In steps S2619 and S2632, the charge processor 23 a of the retrievingserver 23 and the account charge server 24 perform the chargeprocessing. Further, the charge processing is similar to that in theseparating service in FIG. 118 and FIG. 120 and thus, explanationthereof will be omitted. Further, the charge processing in steps S2619and S2632 is the charge processing relating to the service fee fortransmitting the retrieved image. Further, in step S2515, if all are theimages for the retrieving conditions, the processing of step S2616 isskipped.

In step S2603, for example, although ‘house’, ‘cloud’ and ‘face’ arespecified as the retrieving conditions by the client computer 27-4 whenhouse 2181 and cloud 2071 are displayed and face 2201 is not displayedon the image as shown in display 27 a-4, the image is not the desiredimage and then the procedure proceeds to the step S2605 and it isnotified to the retrieving server 23 that it is not the desired imageand then the process ends.

At this time, in step S2618, it is notified to the retrieving server 23that it is not the desired image and the process ends.

In this case, the service fee for the retrieving processing is charged,but the service fee for transmitting the retrieved image is not paid.

According to above description, it is possible to improve the speed ofimage processing by distributing the image processing to a plurality ofserver on the network and it is also possible to reduce cost for theservice by providing only the processing requested by a user be.

Further, following inventions may be considered.

That is, the present invention may relate to a communication apparatusand method and a communication system and more specifically, acommunication apparatus and method and a communication system capable ofimproving the processing speed by distributing the image processing to aplurality of servers on a network to perform it, reducing the cost forthe service by making it possible to perform only the processes desiredby a user by means of the distributed processing, in regard to thedifferences between the real world and the signal detected by a sensor,and regard to the called out images and previous image data which areutilized as needed even without storing it in its own communicationapparatus.

The techniques of detecting events in the real world with a sensor andprocessing the sampling data output from the image sensor is widelyemployed. However, for example, in the image obtained by taking pictureof a body moving in front of a predetermined background being stationarywith a video camera, when moving speed of the body is relatively fast,the motion blur occurs.

As a technique for encrypting the aforementioned image in which themotion blur is present, a burying, method for not recognizing theencrypted image in a flat portion of the image or a method for buryinginformation using correlation of the image have been suggested.

Further, when a body is moving in front of the stationary background,the motion blur occurs when the image of the moving body is mixed by itsown motion and also by the mixture of the background image and the imageof moving body. However, conventionally, the processing corresponding tomixture of the background image and the image of moving body was notconsidered.

Further, in elimination or adjustment of the motion blur in theaforementioned image, after adjusting the shutter, the motion blur maybe tolerated to certain extent to use the motion blur for visual effect.However, in general, when the motion blur is significant, Wiener filteris used for eliminating the motion blur.

Also, a conventional user printed out the aforementioned image as neededthrough selection of the images taken by the user using his owncomputer, etc.

For this reason, the image obtained by separating, synthesizing orcorrecting an image on the basis of the mixture of images is not widelyused except for on the user's computer.

Further, when an image of body in which the motion blur occurred andother background image are combined to be synthesized, conventionalusers obtained the desired background image through a number of searchesfor being input into the user's computer to be displayed.

For this reason, in retrieving images as described above, it poses aproblem that until the desired image was put into the user computer, thesame processing was repeated.

Also, in the techniques for synthesizing the desired images via thenetwork, these synthesized images have been generated by overlapping andjoining a plurality of pre-existing images such as specified imagesobtained from images stored in the storage server, etc. on the networkor images taken with a digital still camera.

By the way, in order to synthesize images by using the aforementionedtechniques, it is premised that images which are bases for synthesizingare previously prepared. However, since there are various know-hows theimage pickup techniques, it is difficult to obtain or generate imagesthemselves that are satisfactory bases. For this reason, there is aproblem that a user cannot easily obtain or generate desired imagedatabases in generating synthesized image.

Further, in consideration of the above problem, if a user could obtainthe desired images, the images having large data capacity must be storedin the user computer without being erased because it is not easy toobtain them again. In this case, it is suggested to perform the optimalencoding in accordance with the feature amounts by separating the imagedata into the feature amounts of the object and the mixture ratio.However, in general, since the feature amounts of the image object andthe mixture ratio have many differences such that the encoding cannot beperformed in the optimal state, there are problems in that the optimalencoding cannot be implemented and the image data cannot be storedefficiently.

In consideration of the foregoing situations, the present inventionmakes it possible to encrypt images using mixed images; eliminate motionblurs by establishing and adjusting the mixture of motion blur to obtainclear images and to correct images more naturally by synthesizing boththe image in which the motion blur is adjusted and the background image;use effectively via the network the images obtained by separating,synthesizing or correcting images on the basis of utilizing mixture ofimages; and retrieve via the network images obtained by separating,synthesizing or correcting images on the basis of utilizing mixture ofimages; call out and use as needed pre-existing image data withoutstoring the image data in the user computer by storing the images in apredetermined storage server and reading out and utilizing images asneeded by using positional information of the storage server.

A 1^(st) system of the present invention is characterized in that afirst communication apparatus comprises: a requested information inputmeans for inputting requested information of a user of a plurality ofencoded data which are generated on the basis of a predetermined imagedata that is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; a requestedinformation transmitting means for transmitting the requestedinformation which is input by the requested information input means to asecond communication apparatus; and an encoded data receiving means forreceiving the encoded data which are generated on the basis of the imagedata corresponding to the requested information transmitted from thesecond communication apparatus, and the second communication apparatuscomprises: a requested information receiving means for receiving therequested information transmitted from the first communicationapparatus; a separating means for separating the predetermined imagedata into a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject; and an encoded data transmitting means for transmitting to thefirst communication apparatus the encoded data which is generated whenthe image is separated into the foreground component image and thebackground component image by the separating means on the basis of therequested information received by the requested information receivingmeans.

Each of the first communication apparatus and the second communicationapparatus may further comprise a first charging means and a secondcharging means for performing charge processing in accordance with therequested information.

The second charging means may generates charge information includinguser ID, the second communication apparatus ID and cost informationcorresponding to the requested information in accordance with therequested information.

The second charging means may perform the charge processing with respectto a financial account of the user on the basis of the chargeinformation by cooperating with the first charging means.

The first charging means of the first communication apparatus mayperform the charge processing by deducting the number of pointscorresponding to the cost information from the number of point of eachuser used in the charge processing.

The encoded data transmitting means may transmit the encoded data onlyto the first communication apparatus after the charge processing hasbeen finished by the first charging means and the second charging means.

The requested information input means may input the predetermined imagedata in addition to the requested information of the user, the requestedinformation transmitting means may transmit the predetermined image datain addition to the requested information to the second communicationapparatus, the requested information receiving means may receive thepredetermined image data in addition to the requested informationtransmitted from the first communication apparatus, the separating meansmay separate the predetermined image data into a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object, and the encoded datatransmitting means may transmit the encoded data as significantinformation generated when the separating means separates thepredetermined image into the foreground component image and thebackground component image on the basis of the predetermined image dataand the requested information received by the requested informationreceiving means, only to the first communication apparatus after thecharge processing has been finished by the first charging means and thesecond charging means.

The second communication apparatus may further comprise an areainformation generating means for generating area information discerningany of a foreground area having foreground object componentsconstituting a foreground object of the predetermined image data, abackground area having background object components constituting abackground object of the predetermined image data and a mixed area inwhich the foreground area and the background area are mixed, wherein thesignificant information includes area information, and wherein theencoded data transmitting means may transmit the encoded data as areainformation generated by area information generating means when theseparating means separates the predetermined image into the foregroundcomponent image and the background component image on the basis of thepredetermined image data and the requested information transmitted bythe requested information input means, only to the first communicationapparatus after the charge processing has been finished by the firstcharging means and the second charging means.

The second communication apparatus may further comprise a mixture ratiogenerating means for generating a mixture ratio indicating a ratio inwhich the foreground area and the background area are mixed in the mixedarea of the predetermined image data, wherein the significantinformation includes the mixture ratio, and wherein the encoded datatransmitting means may transmit the encoded data as mixture ratiogenerated by the mixture ratio generating means when the predeterminedimage is separated into the foreground component image and thebackground component image by the separating means on the basis of thepredetermined image data and the requested information received by therequested information receiving means, only to the first communicationapparatus after the charge processing has been finished by the firstcharging means and the second charging means.

The significant information may include the foreground component imageand the background component image, the encoded data transmitting meansmay transmit the requested information received by the requestedinformation receiving means and the encoded data as the foregroundcomponent image and the background component image generated by theseparating means when the predetermined image is separated into theforeground component image and the background component image by theseparating means, only to the first communication apparatus after thecharge processing has been finished by the first charging means and thesecond charging means.

The requested information input means may input an image data IDdiscerning a predetermined image data in addition to the requestedinformation of the user, and the encoded data transmitting means maytransmit the encoded data as significant information generated when theseparating means separates the predetermined image into the foregroundcomponent image and the background component image on the basis of therequested information and the predetermined image data corresponding tothe image data ID only to the first communication apparatus after thecharge processing has been finished by the first charging means and thesecond charging means.

The second communication apparatus may further comprise an areainformation generating means for generating area information discerningany of a foreground area having foreground object componentsconstituting a foreground object of the predetermined image datacorresponding to the image data ID, a background area having backgroundobject components constituting a background object of the predeterminedimage data corresponding to the image data ID and a mixed area in whichthe foreground area and the background area are mixed, wherein thesignificant information includes area information, and wherein theencoded data transmitting means may transmit the encoded data as areainformation generated by area information generating means when theseparating means separates the predetermined image into the foregroundcomponent image and the background component image on the basis of therequested information and the predetermined image data corresponding tothe image data ID, only to the first communication apparatus after thecharge processing has been finished by the first charging means and thesecond charging means.

The second communication apparatus may further comprise a mixture ratiogenerating means for generating a mixture ratio indicating a ratio ofthe mixed area of the predetermined image data corresponding to theimage data ID on the basis of area information, wherein the significantinformation includes the mixture ratio, and the encoded datatransmitting means may transmit the encoded data as mixture ratiogenerated by the mixture ratio generating means when the separatingmeans separates the predetermined image into the foreground componentimage and the background component image on the basis of the requestedinformation and the predetermined image data corresponding to the imagedata ID only to the first communication apparatus after the chargeprocessing has been finished by the first charging means and the secondcharging means.

The significant information may include the foreground component imageand the background component image, the encoded data transmitting meansmay transmit the requested information and the encoded data as theforeground component image and the background component image generatedby the separating means when the predetermined image corresponding tothe image data ID is separated into the foreground component image andthe background component image by the separating means, only to thefirst communication apparatus after the charge processing has beenfinished by the first charging means and the second charging means.

A communication method of the 1^(st) system of the present invention ischaracterized in that a communication method of the first communicationapparatus comprises: a requested information input step of inputtingrequested information of a user from among a plurality of encoded datawhich are generated on the basis of predetermined image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; a requested informationtransmitting step of transmitting the requested information which isinput in the requested information input step to the secondcommunication apparatus; and an encoded data receiving step of receivingthe encoded data which are generated on the basis of the image datacorresponding to the requested information transmitted from the secondcommunication apparatus, and a communication method of the secondcommunication apparatus comprises: a requested information receivingstep of receiving the requested information transmitted from the firstcommunication apparatus; a separating step of separating thepredetermined image data into a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object; and an encoded data transmitting stepof transmitting to the first communication apparatus the encoded datawhich are generated when the image is separated into, the foregroundcomponent image and the background component image in the separatingstep on the basis of the requested information received in the requestedinformation receiving step.

A program of a 2^(nd) recording medium of the present invention ischaracterized in that a program controlling the first communicationapparatus comprises: a requested information input control step ofcontrolling input of requested information of a user from among aplurality of encoded data which are generated on the basis ofpredetermined image data that is obtained by an image pickup elementhaving a predetermined number of pixels, each having a time integrationeffect, and that is comprised of pixel values determined for everypixels in accordance with an amount of light forming an image integratedover time; a requested information transmission control step ofcontrolling transmission of the requested information which is input inthe requested information input control step to the second communicationapparatus; and an encoded data reception control step of controllingreception of the encoded data which are generated on the basis of theimage data corresponding to the requested information transmitted fromthe second communication apparatus, and a program controlling the secondcommunication apparatus comprises: a requested information receptioncontrol step of controlling reception of the requested informationtransmitted from the first communication apparatus; a separation controlstep of controlling separation of the predetermined image data into aforeground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object; and anencoded data transmission control step of controlling transmission ofthe encoded data which are generated when the image is separated intothe foreground component image and the background component image in theseparation control step on the basis of the requested informationreceived in the requested information reception control step to thefirst communication apparatus.

A 2^(nd) program of the present invention is characterized by making acomputer for controlling the first communication apparatus perform: arequested information input control step of controlling input ofrequested information of a user from among a plurality of encoded datawhich are generated on the basis of predetermined image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated overtime; a requested informationtransmission control step of controlling transmission of the requestedinformation which is input in the requested information input controlstep to the second communication apparatus; and an encoded datareception control step of controlling reception of the encoded datawhich are generated on the basis of the image data corresponding to therequested information transmitted from the second communicationapparatus and by making a computer for controlling the secondcommunication apparatus perform: a requested information receptioncontrol step of controlling reception of the requested informationtransmitted from the first communication apparatus; a separation controlstep of controlling separation of the predetermined image data into aforeground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object; and anencoded data transmission control step of controlling transmission ofthe encoded data which are generated when the image is separated intothe foreground component image and the background component image in theseparation control step on the basis of the requested informationreceived in the requested information reception control step, to thefirst communication apparatus.

A 2^(nd) communication apparatus of the present invention ischaracterized by comprising: a requested information input means forinputting requested information of a user from among a plurality ofencoded data which are generated on the basis of predetermined imagedata that is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; a requestedinformation transmitting means for transmitting the requestedinformation which is input by the requested information input means toother communication apparatus; and an encoded data receiving means forreceiving the encoded data which are generated on the basis of the imagedata corresponding to the requested information transmitted from othercommunication apparatus.

A 2^(nd) communication method of the present invention is characterizedby comprising: a requested information input step of inputting requestedinformation of a user from among a plurality of encoded data which aregenerated on the basis of predetermined image data that is obtained byan image pickup element having a predetermined number of pixels, eachhaving a time integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time; a requested informationtransmitting step of transmitting the requested information which isinput in the requested information input step to other communicationapparatus; and an encoded data receiving step of receiving the encodeddata which are generated on the basis of the image data corresponding tothe requested information transmitted from other communicationapparatus.

A program of a 3^(rd) recording medium of the present invention ischaracterized by comprising: a requested information input control stepof controlling input of requested information of a user from among aplurality of encoded data which are generated on the basis ofpredetermined image data that is obtained by an image pickup elementhaving a predetermined number of pixels, each having a time integrationeffect, and that is comprised of pixel values determined for everypixels in accordance with an amount of light forming an image integratedover time; a requested information transmission control step ofcontrolling transmission of the requested information which is input inthe requested information input control step to other communicationapparatus; and an encoded data reception control step of controllingreception of the encoded data which are generated on the basis of theimage data corresponding to the requested information transmitted fromother communication apparatus.

A 3^(rd) program of the present invention is characterized by making acomputer perform: a requested information input control step ofcontrolling input of requested information of a user from among aplurality of encoded data which are generated on the basis ofpredetermined image data that is obtained by an image pickup elementhaving a predetermined number of pixels, each having a time integrationeffect, and that is comprised of pixel values determined for everypixels in accordance with an amount of light forming an image integratedover time; a requested information transmission control step ofcontrolling transmission of the requested information which is input inthe requested information input control step to other communicationapparatus; and an encoded data reception control step of controllingreception of the encoded data which are generated on the basis of theimage data corresponding to the requested information transmitted fromother communication apparatus.

A 3^(rd) communication apparatus of the present invention ischaracterized by comprising: a requested information receiving means forreceiving the requested information transmitted from other communicationapparatus; a separating means for separating the image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time into a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object; and an encoded datatransmitting means for transmitting to other communication apparatus theencoded data which are generated when the image is separated into theforeground component image and the background component image by theseparating means on the basis of the requested information received bythe requested information receiving means.

A 3^(rd) communication method of the present invention is characterizedby comprising: a requested information receiving step of receiving therequested information transmitted from other communication apparatus; aseparating step of separating the image data that is obtained by animage pickup element having a predetermined number of pixels, eachhaving a time integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time into a foreground component imagehaving foreground object components constituting a foreground object anda background component image having background object componentsconstituting a background object; and an encoded data transmitting stepof transmitting to other communication apparatus the encoded data whichis generated when the image is separated into the foreground componentimage and the background component image in the separating step on thebasis of the requested information received in the requested informationreceiving step.

A program of a 4^(th) recording medium of the present invention ischaracterized by comprising: a requested information receiving step ofreceiving the requested information transmitted from other communicationapparatus; a separating step of separating the image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time into a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object; and an encoded datatransmitting step of transmitting to other communication apparatus theencoded data which is generated when the image is separated into theforeground component image and the background component image in theseparation control step on the basis of the requested informationreceived in the requested information reception control step.

A 4^(th) program of the present invention is characterized by making acomputer perform: a requested information receiving step of receivingthe requested information transmitted from other communicationapparatus; a separating step of separating the image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time into a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object; and an encoded datatransmitting step of transmitting to other communication apparatus theencoded data which is generated when the image is separated into theforeground component image and the background component image in theseparation control step on the basis of the requested informationreceived in the requested information reception control step.

A 2^(nd) communication system of the present invention is characterizedin that the first communication apparatus comprises: a requestedinformation input means for inputting requested information of a userfrom among a plurality of encoded data which are generated on the basisof predetermined image data that is obtained by an image pickup elementhaving a predetermined number of pixels, each having a time integrationeffect, and that is comprised of pixel values determined for everypixels in accordance with an amount of light forming an image integratedover time; a requested information transmitting means for transmittingthe requested information which is input by the requested informationinput means to one of the second communication apparatus to the fourthcommunication apparatus; and an encoded data receiving means forreceiving the encoded data which are generated on the basis of the imagedata corresponding to the requested information transmitted from one ofthe second communication apparatus to the fourth communicationapparatus, the second communication apparatus comprises: the firstrequest information receiving means for receiving the requestedinformation transmitted from the first communication apparatus; aseparating means for separating the predetermined image data into aforeground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object; and thefirst encoded data transmitting means for transmitting to the firstcommunication apparatus the foreground component image and thebackground component image which are separated from the image by theseparating means on the basis of the requested information received bythe first request information receiving means as the encoded data, thethird communication apparatus comprises: the second request informationreceiving means for receiving the requested information transmitted fromthe first communication apparatus; an area information generating meansfor generating area information discerning any of a foreground areahaving foreground object components constituting a foreground object ofthe predetermined image data, a background area having background objectcomponents constituting a background object of the predetermined imagedata and a mixed area in which the foreground area and the backgroundarea are mixed; and the second encoded data transmitting means fortransmitting to the first communication apparatus area informationgenerated by area information generating means on the basis of therequested information received by the second request informationreceiving means as the encoded data, and the fourth communicationapparatus comprises: the third request information receiving means forreceiving the requested information transmitted from the firstcommunication apparatus; a mixture ratio generating means for generatinga mixture ratio indicating a ratio in which the foreground area and thebackground area are mixed in the mixed area of the predetermined imagedata; and the third encoded data transmitting means for transmitting themixture ratio generated by the mixture ratio generating means to thefirst communication apparatus on the basis of the requested informationreceived by the requested information receiving means as encoded data.

A communication method of the 2^(nd) system of the present invention ischaracterized in that a communication method of the first communicationapparatus comprises: a requested information input step of inputtingrequested information of a user from among a plurality of encoded datawhich are generated on the basis of predetermined image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; a requested informationtransmitting step of transmitting the requested information which isinput in the requested information input step to one of the secondcommunication apparatus to the fourth communication apparatus; and anencoded data receiving step of receiving the encoded data which aregenerated on the basis of the image data corresponding to the requestedinformation transmitted from one of the second communication apparatusto the fourth communication apparatus, a communication method of thesecond communication apparatus comprises: the first request informationreceiving step of receiving the requested information transmitted fromthe first communication apparatus; a separating step of separating thepredetermined image data into a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object; and the first encoded datatransmitting step of transmitting to the first communication apparatusthe foreground component image and the background component image whichare separated from the image in the separating step on the basis of therequested information received in the first request informationreceiving step as the encoded data, a communication method of the thirdcommunication apparatus comprises; the second request informationreceiving step of receiving the requested information transmitted fromthe first communication apparatus; an area information generating stepof generating area information discerning any of a foreground areahaving foreground object components constituting a foreground object ofthe predetermined image data, a background area having background objectcomponents constituting a background object of the predetermined imagedata and a mixed area in which the foreground area and the backgroundarea are mixed; and the second encoded data transmitting step oftransmitting to the first communication apparatus area informationgenerated in area information generating step on the basis of therequested information received in the second request informationreceiving step as the encoded data, and a communication method of thefourth communication apparatus comprises: the third request informationreceiving step of receiving the requested information received from thefirst communication apparatus; a mixture ratio generating step ofgenerating a mixture ratio indicating a ratio in which the foregroundarea and the background area are mixed in the mixed area of thepredetermined image data; and the third encoded data transmitting stepof transmitting the mixture ratio generated in the mixture ratiogenerating step to the first communication apparatus on the basis of therequested information received in the requested information receivingstep as the encoded data.

A program of a 5^(th) recording medium of the present invention ischaracterized in that a program controlling the first communicationapparatus comprises: a requested information input control step ofcontrolling input of requested information of a user from among aplurality of encoded data which are generated on the basis ofpredetermined image data that is obtained by an image pickup elementhaving a predetermined number of pixels, each having a time integrationeffect, and that is comprised of pixel values determined for everypixels in accordance with an amount of light forming an image integratedover time; a requested information transmission control step ofcontrolling transmission of the requested information which is input inthe requested information input control step to one of the secondcommunication apparatus to the fourth communication apparatus; and anencoded data reception control step of controlling reception of theencoded data which are generated on the basis of the image datacorresponding to the requested information transmitted from one of thesecond communication apparatus to the fourth communication apparatus, aprogram controlling the second communication apparatus comprises: thefirst request information reception control step of controllingreception of the requested information transmitted from the firstcommunication apparatus; a separation control step of controllingseparation of the predetermined image data into a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object; and the first encoded datatransmission control step of controlling transmission of the foregroundcomponent image and the background component image which are separatedfrom the image in the separation control step on the basis of therequested information received in the first request informationreception step as the encoded data to the first communication apparatus,program controlling the third communication apparatus comprises; thesecond request information reception control step of controllingreception of the requested information transmitted from the firstcommunication apparatus; an area information generation control step ofcontrolling generation of area information discerning any of aforeground area having foreground object components constituting aforeground object of the predetermined image data, a background areahaving background object components constituting a background object ofthe predetermined image data and a mixed area in which the foregroundarea and the background area are mixed; and the second encoded datatransmission control step of controlling transmission of areainformation generated in area information generation control step on thebasis of the requested information received in the second requestinformation reception control step as the encoded data to the firstcommunication apparatus, and a program controlling the fourthcommunication apparatus comprises: the third request informationreception control step of controlling reception of the requestedinformation received from the first communication apparatus; a mixtureratio generation control step of controlling generation of a mixtureratio indicating a ratio in which the foreground area and the backgroundarea are mixed in the mixed area of the predetermined image data; andthe third encoded data transmission control step of controllingtransmission of the mixture ratio generated in the mixture ratiogeneration control step on the basis of the requested informationreceived in the requested information reception control step to thefirst communication apparatus as the encoded data.

A 5^(th) program of the present invention is characterized by making acomputer for controlling the first communication apparatus perform: arequested information input control step of controlling input ofrequested information of a user from among a plurality of encoded datawhich are generated on the basis of predetermined image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; a requested informationtransmission control step of controlling transmission of the requestedinformation which is input in the requested information input step toone of the second communication apparatus to the fourth communicationapparatus; and an encoded data reception control step of controllingreception of the encoded data which are generated on the basis of theimage data corresponding to the requested information transmitted fromone of the second communication apparatus to the fourth communicationapparatus, by making a computer for controlling the second communicationapparatus perform: the first request information reception control stepof controlling reception of the requested information transmitted fromthe first communication apparatus; a separation control step ofcontrolling separation of the predetermined image data into a foregroundcomponent image having foreground object components constituting aforeground object and a background component image having backgroundobject components constituting a background object; and the firstencoded data transmission control step of controlling transmission ofthe foreground component image and the background component image whichare separated from the image in the separation control step on the basisof the requested information received in the first request informationreception control step as the encoded data to the first communicationapparatus, by making a computer for controlling the third communicationapparatus perform; the second request information reception control stepof controlling reception of the requested information transmitted fromthe first communication apparatus; an area information generationcontrol step of controlling generation of area information discerningany of a foreground area having foreground object componentsconstituting a foreground object of the predetermined image data, abackground area having background object components constituting abackground object of the predetermined image data and a mixed area inwhich the foreground area and the background area are mixed; and thesecond encoded data transmission control step of controllingtransmission of area information generated in area informationgeneration control step on the basis of the requested informationreceived in the second request information reception control step as theencoded data to the first communication apparatus, and by making acomputer for controlling the fourth communication apparatus perform: thethird request information reception control step of controllingreception of the requested information received from the firstcommunication apparatus; a mixture ratio generation control step ofcontrolling generation of a mixture ratio indicating a ratio in whichthe foreground area and the background area are mixed in the mixed areaof the predetermined image data; and the third encoded data transmissioncontrol step of controlling transmission of the mixture ratio generatedin the mixture ratio generation control step on the basis of therequested information received in the requested information receptioncontrol step as the encoded data to the first communication apparatus.

A 4^(th) communication apparatus of the present invention ischaracterized by comprising: a requested information input means forinputting the requested information of user; a synthesizing means forsynthesizing a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject of predetermined image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time, to generate a synthesized image; and a synthesizedimage output means for outputting the synthesized image generated by thesynthesizing means.

The synthesized image output means may output the synthesized image tothe communication apparatus of the user.

The requested information input means may input a predeterminedforeground component image, a predetermined background component imageand the significant information used when synthesizing the predeterminedforeground component image and the predetermined background componentimage along with the requested information of the user, and thesynthesizing means may synthesize the predetermined foreground componentimage and the predetermined background component image input by therequested information input means along with the requested informationon the basis of the significant information and generate the synthesizedimage.

The requested information input means may input a predeterminedforeground component image, a predetermined background component imageand the mixture ratio of the mixed area where the predeterminedforeground component image and the predetermined background componentimage are mixed as significant information used when synthesizing thepredetermined foreground component image and the predeterminedbackground component image along with the requested information of theuser, and the synthesizing means may synthesize the predeterminedforeground component image and the predetermined background componentimage input by the requested information input means along with therequested information on the basis of the mixture ratio as significantinformation and generate the synthesized image.

The requested information input means may input a predeterminedforeground component image, a predetermined background component image,and amount of movement and the moving direction of the foregroundcomponent image as significant information used when synthesizing thepredetermined foreground component image and the predeterminedbackground component image along with the requested information of theuser, and the synthesizing means may adjust of the motion blur of thepredetermined foreground component image input by the requestedinformation input means along with the requested information on thebasis of amount of movement and the moving direction of the foregroundcomponent image as the significant information, synthesize thepredetermined foreground component image of which motion blur isadjusted and the predetermined background component image, and generatethe synthesized image.

The requested information input means may input a predeterminedforeground component image, a predetermined background component imageand initial positional information, amount of movement and movingdirection of the foreground component image as significant informationused when synthesizing the predetermined foreground component image andthe predetermined background component image along with the requestedinformation of the user, the synthesizing means may adjust the motionblur of the predetermined foreground component image input by therequested information input means along with the requested informationon the basis of amount of movement and the moving direction assignificant information, calculate the mixture ratio, synthesize thepredetermined foreground component image where motion blur is adjustedand the predetermined background component image using the calculatedmixture ratio on the basis of the initial positional information, amountof movement and the moving direction of the foreground component imageas significant information, and then generate the synthesized image.

The requested information input means may input a foreground componentimage ID discerning a predetermined foreground component image, abackground component image ID discerning a predetermined backgroundcomponent image and significant information along with the requestedinformation of the user, and the synthesizing means may synthesize theforeground component image corresponding to the foreground componentimage ID and the background component image corresponding to thebackground component image ID on the basis of the significantinformation in accordance with the foreground component image IDdiscerning the predetermined foreground component image, the backgroundcomponent image ID discerning the predetermined background componentimage, and significant information along with the requested informationof the user, and generate the synthesized image.

A charging means for performing charge processing in accordance with therequested information may be further comprised.

The charging means may generates charge information including user ID,the communication apparatus ID and cost information corresponding to therequested information in accordance with the requested information.

The charging means may perform the charge processing with respect to afinancial account of the user on the basis of the charge information.

The charging means may perform the charge processing by deducting thenumber of points corresponding to the cost information from the numberof point of each user used in the charge processing and corresponding tocash.

The output means may output the synthesized image to the communicationapparatus of the user having finished the charge processing after thecharge processing has been finished by the charging means.

A 4^(th) communication method of the present invention is characterizedby comprising: a requested information input step of inputting therequested information of user; a synthesizing step of synthesizing aforeground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object ofpredetermined image data that is obtained by an image pickup elementhaving a predetermined number of pixels, each having a time integrationeffect, and that is comprised of pixel values determined for everypixels in accordance with an amount of light forming an image integratedover time in accordance with request information and generating asynthesized image; and a synthesized image output step of outputting thesynthesized image generated in the synthesizing step.

A program of a 6^(th) recording medium of the present invention ischaracterized by comprising: a requested information input control stepof controlling input of the requested information of user; asynthesization control step of controlling synthesization of aforeground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object ofpredetermined image data that is obtained by an image pickup elementhaving a predetermined number of pixels, each having a time integrationeffect, and that is comprised of pixel values determined for everypixels in accordance with an amount of light forming an image integratedover time in accordance with request information, to generate asynthesized image; and a synthesized image output control step ofcontrolling output of the synthesized image generated in thesynthesization control step.

A 6^(th) program of the present invention is characterized by making acomputer perform: a requested information input control step ofcontrolling input of the requested information of user; a synthesizationcontrol step of controlling synthesization of a foreground componentimage having foreground object components constituting foreground objectand a background component image having background object componentsconstituting a background object of predetermined image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time in accordance with requestinformation, to generate a synthesized image; and a synthesized imageoutput control step of controlling output of the synthesized imagegenerated in the synthesization control step.

A 3^(rd) communication system of the present invention is characterizedin that the first communication apparatus comprises: a requestedinformation input means for inputting request information of a user; arequested information transmitting means for transmitting the requestedinformation which is input by the requested information input means tothe second communication apparatus; and a synthesized data receivingmeans for receiving the synthesized data transmitted from the secondcommunication apparatus in accordance with the requested information,and the second communication apparatus comprises: a requestedinformation receiving means for receiving the requested informationtransmitted from the first communication apparatus; a synthesizing meansfor synthesizing a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject of predetermined image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time in accordance with the requested information andgenerating the synthesized image; and a synthesized image transmittingmeans for transmitting the synthesized image generated by thesynthesizing means to the first communication apparatus.

The requested information input means may input a predeterminedforeground component image, a predetermined background component imageand the significant information used when synthesizing the predeterminedforeground component image and the predetermined background componentimage along with the requested information of the user, and thesynthesizing means may synthesize the predetermined foreground componentimage and the predetermined background component image input by therequested information input means along with the requested informationon the basis of the significant information and generate the synthesizedimage.

The requested information input means may input a predeterminedforeground component image, a predetermined background component imageand the mixture ratio of the mixed area where the predeterminedforeground component image and the predetermined background componentimage are mixed as significant information used when synthesizing thepredetermined foreground component image and the predeterminedbackground component image along with the requested information of theuser, and the synthesizing means may synthesize the predeterminedforeground component image and the predetermined background componentimage input by the requested information input means along the requestedinformation on the basis of the mixture ratio as significant informationand generate the synthesized image.

The requested information input means may input a predeterminedforeground component image, a predetermined background component image,and amount of movement and moving direction of the foreground componentimage as significant information used when synthesizing thepredetermined foreground component image and the predeterminedbackground component image along with the requested information of theuser, and the synthesizing means may adjust of the predeterminedforeground component image input by the requested information inputmeans along with the requested information on the basis of amount ofmovement and the moving direction of the foreground component image assignificant information, synthesize the predetermined foregroundcomponent image in which the motion blur is adjusted and thepredetermined background component image and generate the synthesizedimage.

The requested information input means may input a predeterminedforeground component image, a predetermined background component imageand initial positional information, amount of movement and movingdirection of the foreground component image as significant informationused when synthesizing the predetermined foreground component image andthe predetermined background component image along with the requestedinformation of the user, and the synthesizing means may adjust themotion blur of the predetermined foreground component image input by therequested Information input means along with the requested informationon the basis of amount of movement and the moving direction assignificant information, calculate the mixture ratio, synthesize thepredetermined foreground component image where motion blur is adjustedand the predetermined background component image using the calculatedmixture ratio on the basis of the initial positional information, amountof movement and the moving direction of the foreground component imageas significant information, and then generate the synthesized image.

The requested information input means may input a foreground componentimage ID discerning a predetermined foreground component image, abackground component image ID discerning a predetermined backgroundcomponent image and significant information along with the requestedinformation of the user, and the synthesizing means may synthesize theforeground component image corresponding to the foreground componentimage ID and the background component image corresponding to thebackground component image ID on the basis of the significantinformation in accordance with the foreground component image IDdiscerning the predetermined foreground component image, the backgroundcomponent image ID discerning the predetermined background componentimage, and significant information along with the requested informationof the user, and generate the synthesized image.

The charging means for performing charge processing in accordance withthe requested information may be further comprised.

The charging means may generates charge information including user ID,the communication apparatus ID and charge information including the costinformation corresponding to the requested information in accordancewith the requested information.

The charging means may perform the charge processing with respect to afinancial account of the user on the basis of the charge information.

The charging means may perform the charge processing by deducting thenumber of points corresponding to the cost information from the numberof points of each user used in the charge processing.

The output means may output the synthesized image to the communicationapparatus of the user having finished the charge processing after thecharge processing has been finished by the charging means.

A communication method of the 3^(rd) communication system of the presentinvention is characterized in that a communication method of the firstcommunication apparatus comprises: a requested information input step ofinputting request information of a user; a requested informationtransmitting step of transmitting the requested information which isinput in the requested information input step to the secondcommunication apparatus; and a synthesized data receiving step ofreceiving the synthesized data transmitted from the second communicationapparatus in accordance with the requested information, and acommunication method of the second communication apparatus comprises: arequested information receiving step of receiving the requestedinformation transmitted from the first communication apparatus; asynthesizing step of synthesizing a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting background object of predetermined image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time in accordance with therequested information and generating the synthesized image; and asynthesized image transmitting step of transmitting the synthesizedimage generated in the synthesizing step to the first communicationapparatus.

A program of a 7^(th) recording medium of the present invention ischaracterized in that a program controlling the first communicationapparatus comprises: a requested information input control step ofcontrolling input of request information of a user; a requestedinformation transmission control step of controlling transmission of therequested information which is input in the requested information inputcontrol step to the second communication apparatus; and a synthesizeddata reception control step of controlling reception of the synthesizeddata transmitted from the second communication apparatus in accordancewith the requested information, and a program controlling the secondcommunication apparatus comprises: a requested information receptioncontrol step of controlling reception of the requested informationtransmitted from the first communication apparatus; a synthesizationcontrol step of controlling synthesization of a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object of predetermined image datathat is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time in accordancewith the requested information and generating the synthesized image; anda synthesized image transmission control step of controllingtransmission of the synthesized image generated in the synthesizing stepto the first communication apparatus.

A 7^(th) program of the present invention is characterized by making acomputer for controlling the first communication apparatus perform: arequested information input control step of controlling input of requestinformation of a user; a requested information transmission control stepof controlling transmission of the requested information which is inputin the requested information input control step to the secondcommunication apparatus; and a synthesized data reception control stepof controlling reception of the synthesized data transmitted from thesecond communication apparatus in accordance with the requestedinformation, and by making a computer for controlling the secondcommunication apparatus perform: a requested information receptioncontrol step of controlling reception of the requested informationtransmitted from the first communication apparatus; a synthesizationcontrol step of controlling synthesization of a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object of predetermined image datathat is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time in accordancewith the requested information and generating the synthesized image; anda synthesized image transmission control step of controllingtransmission of the synthesized image generated in the synthesizationcontrol step to the first communication apparatus.

A 5^(th) communication apparatus of the present invention ischaracterized by comprising: a requested information input means forinputting request information of a user; a requested informationtransmitting means for transmitting the requested information which isinput by the requested information input means to other communicationapparatus; and a synthesized data receiving means for receiving thesynthesized data transmitted from other communication apparatus inaccordance with the requested information.

A 5^(th) communication method of the present invention is characterizedby comprising: a requested information input step of inputting requestinformation of a user; a requested information transmitting step oftransmitting the requested information which is input in the requestedinformation input step to other communication apparatus; and asynthesized data receiving step of receiving the synthesized datatransmitted from other communication apparatus in accordance with therequested information.

A program of an 8^(th) recording medium of the present invention ischaracterized by comprising: a requested information input control stepof controlling input of request information of a user; a requestedinformation transmission control step of controlling transmission of therequested information which is input in the requested information inputcontrol step to other communication apparatus; and a synthesized datareception control step of controlling reception of the synthesized datatransmitted from other communication apparatus in accordance with therequested information.

An 8^(th) program of the present invention is characterized by making acomputer perform: a requested information input control step ofcontrolling input of request information of a user; a requestedinformation transmission control step of controlling transmission of therequested information which is input in the requested information inputcontrol step to other communication apparatus; and a synthesized datareception control step of controlling reception of the synthesized datatransmitted from other communication apparatus in accordance with therequested information.

A 6^(th) communication apparatus of the present invention ischaracterized by comprising: a requested information receiving means forreceiving the requested information transmitted from other communicationapparatus; a synthesizing means for synthesizing a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object of predetermined image datathat is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time in accordancewith the requested information and generating the synthesized image; anda synthesized image transmitting means for transmitting the synthesizedimage generated by the synthesizing means to other communicationapparatus.

A 6^(th) communication method of the present invention is characterizedby comprising: a requested information receiving step of receiving therequested information transmitted from other communication apparatus; asynthesizing step of synthesizing a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object of predetermined image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time in accordance with therequested information and generating the synthesized image; and asynthesized image transmitting step of transmitting the synthesizedimage generated in the synthesizing step to other communicationapparatus.

A program of a 9^(th) recording medium of the present invention ischaracterized by comprising: a requested information reception controlstep of controlling reception of the requested information transmittedfrom other communication apparatus; a synthesization control step ofcontrolling synthesization of a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object of predetermined image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time in accordance with therequested information and generating the synthesized image; and asynthesized image transmission control step of controlling transmissionof the synthesized image generated in the synthesization control step toother communication apparatus.

A 9^(th) program of the present invention is characterized by making acomputer perform: a requested information reception control step ofcontrolling reception of the requested information transmitted fromother communication apparatus; a synthesization control step ofcontrolling synthesization of a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object of predetermined image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time in accordance with therequested information and generating the synthesized image; and asynthesized image transmission control step of controlling transmissionof the synthesized image generated in the synthesization control step toother communication apparatus.

A 7^(th) communication apparatus of the present invention ischaracterized by comprising: a requested information input means forinputting the requested information of a user; a mixture ratioestimating means for estimating a mixture ratio indicating a ratio inwhich a foreground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object of theimage data that is obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values=determined for every pixels inaccordance with an amount of light forming an image integrated over timeare mixed on the basis of the requested information; a separating meansfor separating the image data into the foreground component image andthe background component image on the basis of the mixture ratio; asynthesizing means for synthesizing the foreground component imageseparated by the separating means and any background component image, orthe background component image separated by the separating means and anyforeground component image in a predetermined mixture ratio andgenerating the synthesized image; and a output means for outputting thesynthesized image synthesized by the synthesizing means.

The requested information input means may input amount of motion bluradjustment for adding motion blur to the foreground component image inaddition to the requested information of the user and further comprise amotion blur adding means which adds motion blur corresponding to amountby which motion blur is adjusted to the foreground component image, andthe synthesizing means may synthesize the foreground component image inwhich motion blur is added by the motion blur adding means and anybackground component image in a predetermined mixture ratio and generatethe synthesized image.

The requested information input means may also input any backgroundcomponent image in addition to the requested information of the user,and the synthesizing means may synthesize the foreground component imageand any background component image in a ratio estimated by the mixtureratio estimating means and generate the synthesized image.

The charging means for performing charge processing in accordance withthe requested information may be further comprised.

The charging means may generates charge information including user ID,the communication apparatus ID and cost information corresponding to therequested information in accordance with the requested information.

The charging means may perform the charge processing with respect to afinancial account of the user on the basis of the charge information.

The charging means may perform the charge processing by deducting thenumber of points corresponding to the cost information from the numberof points of each user used in the charge processing.

The output means may output the synthesized image to the communicationapparatus of the user having finished the charge processing after thecharge processing has been finished by the charging means.

The requested information input means may input a predetermined imagedata in addition to the requested information of the user, and theoutput means may output the image synthesized by the synthesizing meansto the communication apparatus of the user having finished the chargeprocessing after the charge processing has been finished by the chargingmeans.

A 7^(th) communication method of the present invention is characterizedby comprising: a requested information input step of inputting therequested information of a user; a mixture ratio estimating step ofestimating a mixture ratio indicating a ratio in which a foregroundcomponent image having foreground object components constituting aforeground object and a background component image having backgroundobject components constituting a background object of the image datathat is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time are mixed onthe basis of the requested information; a separating step of separatingthe image data into the foreground component image and the backgroundcomponent image on the basis of the mixture ratio; a synthesizing stepof synthesizing the foreground component image separated in theseparating step and any background component image, or the backgroundcomponent image separated in the separating step and any foregroundcomponent image in a predetermined mixture ratio and generating thesynthesized image; and an output step of outputting the synthesizedimage synthesized in the synthesizing step.

A program of a 10^(th) recording medium of the present invention ischaracterized by comprising: a requested information input control stepof controlling input of the requested information of a user; a mixtureratio estimation control step of controlling estimation of a mixtureratio indicating a ratio in which a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object of the image data that is obtained byan image pickup element having a predetermined number of pixels, eachhaving a time integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time are mixed on the basis of therequested information; a separation control step of controllingseparation of the image data into the foreground component image and thebackground component image on the basis of the mixture ratio; asynthesization control step of controlling synthesization of theforeground component image separated in the separation control step andany background component image, or the background component imageseparated in the separation control step and any foreground componentimage in a predetermined mixture ratio and generating the synthesizedimage; and a output control step of controlling output of thesynthesized image synthesized in the synthesization control step.

A 10^(th) program of the present invention is characterized by making acomputer perform: a requested information input control step ofcontrolling input of the requested information of a user; a mixtureratio estimation control step of controlling estimation of a mixtureratio indicating a ratio in which a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object of the image data that is obtained byan image pickup element having a predetermined number of pixels, eachhaving a time integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time are mixed on the basis of therequested information; a separation control step of controllingseparation of the image data into the foreground component image and thebackground component image on the basis of the mixture ratio; asynthesization control step of controlling synthesization of theforeground component image separated in the separation control step andany background component image, or the background component imageseparated in the separation control step and any foreground componentimage in a predetermined mixture ratio and generating the synthesizedimage; and a output control step of controlling output of thesynthesized image synthesized in the synthesization control step.

A 4^(th) communication system of the present invention is characterizedin that the first communication apparatus comprises: a requestedinformation input means for inputting request information of a user; arequested information transmitting means for transmitting the requestedinformation which is input by the requested information input means tothe second transmitting apparatus; and a receiving means for receivingthe synthesized image transmitted from the second communicationapparatus, and the second communication apparatus comprises: a requestedinformation receiving means for receiving the requested informationwhich is transmitted from the first transmitting apparatus; a mixtureratio estimating means for estimating a mixture ratio indicating a ratioin which a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject of the image data that is obtained by an image pickup elementhaving a predetermined number of pixels, each having a time integrationeffect, and that is comprised of pixel values determined for everypixels in accordance with an amount of light forming an image integratedover time are mixed on the basis of the requested information; aseparating means for separating the image data into the foregroundcomponent image and the background component image on the basis of themixture ratio; a synthesizing means for synthesizing the foregroundcomponent image separated by the separating means and any backgroundcomponent image, or the background component image separated by theseparating means and any foreground component image in a predeterminedmixture ratio and generating the synthesized image; and a synthesizedimage transmitting means for transmitting the synthesized imagesynthesized by the synthesizing means to the first communicationapparatus.

The requested information input means may input amount of motion bluradjustment for adding motion blur to the foreground component image inaddition to the requested information of the user, the secondcommunication apparatus may further comprise a motion blur adding meanswhich adds motion blur corresponding to amount by which motion blur isadjusted to the foreground component image, and the synthesizing meansmay synthesize the foreground component image where motion blur is addedby the motion blur adding means and any background component image in apredetermined mixture ratio and generate the synthesized image.

The requested information input means may also input any backgroundcomponent image in addition to the requested information of the user,the synthesizing means may synthesize the foreground component image andany background component image in a mixture ratio estimated by themixture ratio estimating means and generate the synthesized image.

The second communication apparatus may further comprise the chargingmeans for performing charge processing in accordance with the requestedinformation.

The charging means may generates charge information including user ID,the second communication apparatus ID and cost information correspondingto the requested information in accordance with the requestedinformation.

The charging means may perform the charge processing with respect to afinancial account of the user on the basis of the charge information.

The charging means of the second communication apparatus may perform thecharge processing by deducting the number of points corresponding to thecost information from the number of point of each user used in thecharge processing.

The synthesized image transmitting means may transmit the synthesizedimage to the first communication apparatus of the user having finishedthe charge processing after the charge processing has been finished bythe charging means.

The requested information input means may input a predetermined imagedata in addition to the requested information of the user, and theoutput means may output the image synthesized by the synthesizing meansto the first communication apparatus of the user having finished thecharge processing after the charge processing has been finished by thecharging means.

A communication method of the 4^(th) communication system of the presentinvention is characterized by a communication method of the firstcommunication apparatus comprises: a requested information input step ofinputting request information of a user; a requested informationtransmitting step of transmitting the requested information which isinput in the requested information input step to the second transmittingapparatus; and a receiving step of receiving the synthesized imagetransmitted from the second communication apparatus, and a communicationmethod of the second communication apparatus comprises: a requestedinformation receiving step of receiving the requested information whichis transmitted from the first transmitting apparatus; a mixture ratioestimating step of estimating a mixture ratio indicating a ratio inwhich a foreground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object of theimage data that is obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values determined for every pixels inaccordance with an amount of light forming an image integrated over timeare mixed on the basis of the requested information; a separating stepof separating the image data into the foreground component image and thebackground component image on the basis of the mixture ratio; asynthesizing step of synthesizing the foreground component imageseparated in the separating step and any background component image, orthe background component image separated in the separating step and anyforeground component image in a predetermined mixture ratio andgenerating the synthesized image; and a synthesized image transmittingstep of transmitting the synthesized image synthesized in thesynthesizing step to the first communication apparatus.

A program of an 11^(th) recording medium of the present invention ischaracterized in that a program controlling the first communicationapparatus comprises: a requested information input control step ofcontrolling input of the requested information of a user; a requestedinformation transmission control step of controlling transmission of therequested information which is input in the requested information inputcontrol step to the second transmitting apparatus; and a receptioncontrol step of controlling reception of the synthesized imagetransmitted from the second communication apparatus, and a programcontrolling the second communication apparatus comprises: a requestedinformation reception control step of controlling reception of therequested information which is transmitted from the first transmittingapparatus; a mixture ratio estimation control step of controllingestimation of a mixture ratio indicating a ratio in which a foregroundcomponent image having foreground object components constituting aforeground object and a background component image having backgroundobject components constituting a background object of the image datathat is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time are mixed onthe basis of the requested information; a separation control step ofcontrolling separation of the image data into the foreground componentimage and the background component image on the basis of the mixtureratio; a synthesization control step of controlling synthesization ofthe foreground component image separated in the separation control stepand any background component image, or the background component imageseparated in the separation control step and any foreground componentimage in a predetermined mixture ratio and generating the synthesizedimage; and a synthesized image transmission control step of controllingtransmission of the synthesized image synthesized in the synthesizationcontrol step to the first communication apparatus.

An 11^(th) program of the present invention is characterized by making acomputer for controlling the first communication apparatus perform: arequested information input control step of controlling input of therequested information of a user; a requested information transmissioncontrol step of controlling transmission of the requested informationwhich is input in the requested information input control step to thesecond transmitting apparatus; and a reception control step ofcontrolling reception of the synthesized image transmitted from thesecond communication apparatus, and by making a computer for controllingthe second communication apparatus perform: a requested informationreception control step of controlling reception of the requestedinformation which is transmitted from the first transmitting apparatus;a mixture ratio estimation control step of controlling estimation of amixture ratio indicating a ratio in which a foreground component imagehaving foreground object components constituting a foreground object anda background component image having background object componentsconstituting a background object of the image data that is obtained byan image pickup element having a predetermined number of pixels, eachhaving a time integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time are mixed on the basis of therequested information; a separation control step of controllingseparation of the image data into the foreground component image and thebackground component image on the basis of the mixture ratio; asynthesization control step of controlling synthesization of theforeground component image separated in the separation control step andany background component image, or the background component imageseparated in the separation control step and any foreground componentimage in a predetermined mixture ratio and generating the synthesizedimage; and a synthesized image transmission control step of controllingtransmission of the synthesized image synthesized in the synthesizationcontrol step to the first communication apparatus.

An 8^(th) communication apparatus of the present invention ischaracterized by comprising: a requested information input means forinputting request information of a user; a requested informationtransmitting means for transmitting the requested information which isinput by the requested information input means to other communicationapparatus; and a receiving means for receiving the synthesized imagetransmitted from other communication apparatus.

An 8^(th) communication method of the present invention is characterizedby comprising: a requested information input step of inputting requestinformation of a user; a requested information transmitting step oftransmitting the requested information which is input in the requestedinformation input step to other communication apparatus; and a receivingstep of receiving the synthesized image transmitted from othercommunication apparatus.

A program of a 12^(th) recording medium of the present invention ischaracterized by comprising: a requested information input control stepof controlling input of request information of a user; a requestedinformation transmission control step of controlling transmission of therequested information which is input in the requested information inputcontrol step to other communication apparatus; and a reception controlstep of controlling reception of the synthesized image transmitted fromother communication apparatus.

A 12^(th) program of the present invention is characterized by making acomputer perform: a requested information input control step ofcontrolling input of request information of a user; a requestedinformation transmission control step of controlling transmission of therequested information which is input in the requested information inputcontrol step to other communication apparatus; and a reception controlstep of controlling reception of the synthesized image transmitted fromother communication apparatus.

A 9^(th) communication apparatus of the present invention ischaracterized by comprising: a requested information receiving means forreceiving the requested information which is transmitted from othercommunication apparatus; a mixture ratio estimating means for estimatinga mixture ratio indicating a ratio in which a foreground component imagehaving foreground object components constituting a foreground object anda background component image having background object componentsconstituting a background object of the image data that is obtained byan image pickup element having a predetermined number of pixels, eachhaving a time integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time are mixed on the basis of therequested information; a separating means for separating the image datainto the foreground component image and the background component imageon the basis of the mixture ratio; a synthesizing means for synthesizingthe foreground component image separated by the separating means and anybackground component image, or the background component image separatedby the separating means and any foreground component image in apredetermined mixture ratio and generating the synthesized image; and asynthesized image transmitting means for transmitting the synthesizedimage synthesized by the synthesizing means to other communicationapparatus.

A 9^(th) communication method of the present invention is characterizedby comprising: a requested information receiving step of receiving therequested information which is transmitted from other communicationapparatus; a mixture ratio estimating step of estimating a mixture ratioindicating a ratio in which a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object of the image data that is obtained byan image pickup element having a predetermined number of pixels, eachhaving a time integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time are mixed on the basis of therequested information; a separating step of separating the image datainto the foreground component image and the background component imageon the basis of the mixture ratio; a synthesizing step of synthesizingthe foreground component image separated in the separating step and anybackground component image, or the background component image separatedin the separating step and any foreground component image in apredetermined mixture ratio and generating the synthesized image; and asynthesized image transmitting step of transmitting the synthesizedimage synthesized in the synthesizing step to other communicationapparatus.

A program of a 13^(th) recording medium of the present invention ischaracterized by comprising: a requested information reception controlstep of controlling reception of the requested information which istransmitted from other communication apparatus; a mixture ratioestimation control step of controlling estimation of a mixture ratioindicating a ratio in which a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object of the image data that is obtained byan image pickup element having a predetermined number of pixels, eachhaving a time integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time are mixed on the basis of therequested information; a separation control step of controllingseparation of the image data into the foreground component image and thebackground component image on the basis of the mixture ratio; asynthesization control step of controlling synthesization of theforeground component image separated in the separation control step andany background component image, or the background component imageseparated in the separation control step and any foreground componentimage in a predetermined mixture ratio and generating the synthesizedimage; and a synthesized image transmission control step of controllingtransmission of the synthesized image synthesized in the synthesizationcontrol step to other communication apparatus.

A 13^(th) program of the present invention is characterized by making acomputer perform: a requested information reception control step ofcontrolling reception of the requested information which is transmittedfrom other communication apparatus; a mixture ratio estimation controlstep of controlling estimation of a mixture ratio indicating a ratio inwhich a foreground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object of theimage data that is obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values determined for every pixels inaccordance with an amount of light forming an image integrated over timeare mixed on the basis of the requested information; a separationcontrol step of controlling separation of the image data into theforeground component image and the background component image on thebasis of the mixture ratio; a synthesization control step of controllingsynthesization of the foreground component image separated in theseparation control step and any background component image, or thebackground component image separated in the separation control step andany foreground component image in a predetermined mixture ratio andgenerating the synthesized image; and a synthesized image transmissioncontrol step of controlling transmission of the synthesized imagesynthesized in the synthesization control step to other communicationapparatus.

A 10^(th) communication apparatus of the present invention ischaracterized by comprising; an image ID input means for inputting theID discerning an image requested by a user; a storing means for storingthe image having at least one image of a foreground component imagehaving foreground object components constituting a foreground object anda background component image having background object componentsconstituting a background object separated from image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; a retrieving means forretrieving an image corresponding to an image ID input by the image IDinput means of the images stored in the storing means; an output meansfor outputting the image retrieved by the retrieving means to thecommunication apparatus of the user; and ‘a charging means forperforming charge processing in accordance with the image retrieved bythe retrieving means.

A separating means may be further comprised for separating the imageinto a foreground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object and thestoring means may store the foreground component image and thebackground component image separated from the image by the separatingmeans.

A synthesized image generating means may be further comprised forsynthesizing the foreground component image having foreground objectcomponents constituting a foreground object and the background componentimage having background object components constituting a backgroundobject separated from the image in a predetermined mixture ratio andgenerating the synthesized image and the storing means may store thesynthesized image generated by the synthesized image generating means.

A separating means for separating the image into the foregroundcomponent image having foreground object components constituting aforeground object and the background component image having backgroundobject components constituting a background object and a synthesizedimage generating means for synthesizing the foreground component imageand the background component image separated by the separating means ina predetermined mixture ratio and generating the synthesized image maybe further comprised, and the storing means may store the synthesizedimage generated by the synthesized image generating means.

The charging means may generates charge information including, user ID,the communication apparatus ID and cost information corresponding to theimage retrieved by the retrieving means in accordance with the imageretrieved by the retrieving means.

The charging means may perform the charge processing with respect to afinancial account of the user on the basis of the charge information.

The charging means may perform the charge processing by deducting thenumber of points corresponding to the cost information from the numberof point of each user used in the charge processing.

The output means may output the synthesized image to the firstcommunication apparatus of the user having finished the chargeprocessing after the charge processing has been finished by the chargingmeans.

A 10^(th) communication method of the present invention is characterizedby comprising; an image ID input step of inputting the ID discerning animage requested by a user; a storing step of storing the image comprisedof at least one image of a foreground component image having foregroundobject components constituting a foreground object and a backgroundcomponent image having background object components constituting abackground object separated from image data that is obtained by an imagepickup element having a predetermined number of pixels, each having atime integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time; a retrieving step of retrievingan image corresponding to an image ID input in the image ID input stepof the images stored in the storing step; an output step of outputtingthe image retrieved in the retrieving step to the communicationapparatus of the user; and a charging step of performing chargeprocessing according to the image retrieved in the retrieving step.

A program of a 14^(th) recording medium of the present invention ischaracterized by comprising; an image ID input control step ofcontrolling input of the ID discerning an image requested by a user; astorage control step of controlling storage of the image comprised of atleast one image of a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject separated from image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time; a retrieval control step of controlling retrievalof an image corresponding to an image ID input in the image ID inputcontrol step of the images stored in the storage control step; an outputcontrol step of controlling output of the image retrieved in theretrieval control step to the communication apparatus of the user; and acharge control step of controlling performance of charge processingaccording to the image retrieved in the retrieval control step.

A 14^(th) program of the present invention is characterized by making acomputer perform; an image ID input control step of controlling input ofthe discerning an image requested by a user; a storage control step ofcontrolling storage of the image comprised of at least one image of aforeground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object separatedfrom image data that is obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values determined for every pixels inaccordance with an amount of light forming an image integrated overtime; a retrieval control step of controlling retrieval of an imagecorresponding to an image ID input in the image ID input control step ofthe images stored in the storage control step; an output control step ofcontrolling output of the image retrieved in the retrieval control stepto the communication apparatus of the user; and a charge control step ofcontrolling performance of charge processing according to the imageretrieved in the retrieval control step.

A 5^(th) communication system of the present invention is characterizedin that a first communication apparatus comprises: an image ID inputmeans for inputting an ID discerning an image requested by a user; animage ID transmitting means for transmitting the image ID to a secondcommunication apparatus; and an image receiving means for receiving theimage retrieved according to the image ID, and the second communicationapparatus comprises: an image ID receiving means for receiving the imageID transmitted from the first communication apparatus; a storing meansfor storing the image comprised of at least one image of a foregroundcomponent image having foreground object components constituting aforeground object and a background component image having backgroundobject components constituting a background object separated from imagedata that is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; a retrievingmeans for retrieving the image corresponding to the image ID received bythe image ID receiving means among images stored in the storing means;an output means for outputting the image retrieved by the retrievingmeans to the first communication apparatus; and a charging means forperforming a charge processing according to the image retrieved by theretrieving means.

The second communication apparatus may further comprise a separatingmeans for separating the image into the foreground component imagehaving foreground object components constituting the foreground objectand the background component image having background object componentsconstituting the background object.

The storing means may store the foreground component image and thebackground component image separated by the separating means.

The second communication apparatus may further comprise a synthesizedimage generating means for synthesizing the foreground component imagehaving foreground object components constituting the foreground objectand the background component image having background object componentsconstituting the background object separated from the image andgenerating the synthesized image in a predetermined mixture ratio andthe storing means may store the synthesized image generated by thesynthesized image generating means.

The second communication apparatus may further comprise: a separatingmeans for separating the image into the foreground component imagehaving foreground object components constituting the foreground objectand the background component image having background object componentsconstituting the background object; and a synthesized image generatingmeans for synthesizing the foreground component image having foregroundobject components constituting the foreground object and the backgroundcomponent image having background object components constituting thebackground object separated by the separating means and generating thesynthesized image in a predetermined mixture ratio and the storing meansmay store the synthesized image generated by the synthesized imagegenerating means.

The charging means may generate charge information including a user ID,a second communication apparatus ID and cost information correspondingto the image retrieved by the retrieving means in accordance with theimage retrieved by the retrieving means.

The charging means may perform the charge processing with respect to afinancial account of the user on the basis of the charge information.

The charging means of the second communication apparatus may perform thecharge processing by subtracting the number of points corresponding tothe cost information from the number of points of each user used in thecharge processing.

The output means may output the image to the first communicationapparatus of the user having finished the charge processing after thecharge processing has been finished by the charging means.

A communication method of the 5^(th) communication system of the presentinvention is characterized in that a communication method of the firstcommunication apparatus comprises: an image ID input step of inputtingan ID discerning an image requested by a user; an image ID transmittingstep of transmitting the image ID to the second communication apparatus;and an image receiving step of receiving the image retrieved inaccordance with the image ID, and a communication method of the secondcommunication apparatus comprises: a storing step of storing the imagecomprised of at least one image of a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object separated from the image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; a retrieving step ofretrieving the image corresponding to the image ID received in the imageID receiving step among images stored in the storing step; an outputstep of outputting the image retrieved in the retrieving step to thefirst communication apparatus; and a charging step of performing acharge processing in accordance with the image retrieved in theretrieving step.

A program of a 15^(th) recording medium of the present invention ischaracterized in that a program for controlling the first communicationapparatus comprises: an image ID input control step of controlling inputof an ID discerning an image requested by a user; an image IDtransmission control step of controlling transmission of the image ID tothe second communication apparatus; and an image reception control stepof controlling reception of the image retrieved in accordance with theimage ID, and a program for controlling the second communicationapparatus comprises: a storage control step of controlling storage ofthe image comprised of at least one image of a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object separated from image datathat is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; a retrievalcontrol step of controlling retrieval of the image corresponding to theimage ID received in the image ID reception control step among imagesstored in the storage control step; an output control step ofcontrolling output of the image retrieved in the retrieval control stepto the first communication apparatus; and a charge control step ofcontrolling performance of a charge processing in accordance with theimage retrieved in the retrieval control step.

A 15^(th) program of the present invention is characterized by making acomputer for controlling the first communication apparatus perform: animage ID input control step of controlling input of an ID discerning animage requested by a user; an image ID transmission control step ofcontrolling transmission of the image ID to the second communicationapparatus; and an image reception control step of controlling receptionof the image retrieved in accordance with the image ID, and by making acomputer for controlling the second communication apparatus perform: astorage control step of controlling storage of the image comprised of atleast one image of a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject separated from image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time; a retrieval control step of controlling retrievalof the image corresponding to the image ID received in, the image IDreception control step among images stored in the storage control step;an output control step of controlling output of the image retrieved inthe retrieval control step to the first communication apparatus; and acharge control step of controlling performance of a charge processing inaccordance with the image retrieved in the retrieval control step.

An 11^(th) communication apparatus of the present invention ischaracterized by comprising: an image ID input means for inputting an IDdiscerning an image requested by a user; an image ID transmitting meansfor transmitting the image ID to other communication apparatus; and animage receiving means for receiving the image retrieved correspondinglyto the image ID.

An 11^(th) communication method of the present invention ischaracterized by comprising: an image ID input step of inputting an IDdiscerning an image requested by a user; an image ID transmitting stepof transmitting the image ID to other communication apparatus; and animage receiving step of receiving the image retrieved correspondingly tothe image ID.

A program of a 16^(th) recording medium of the present invention ischaracterized by comprising: an image ID input control step ofcontrolling input of an ID discerning an image requested by a user; animage ID transmission control step of controlling transmission of theimage ID to other communication apparatus; and an image receptioncontrol step of controlling reception of the image retrievedcorrespondingly to the image ID.

A 16^(th) program of the present invention is characterized by making acomputer perform: an image ID input control step of controlling input ofan ID discerning an image requested by a user; an image ID transmissioncontrol step of controlling transmission of the image ID to othercommunication apparatus; and an image reception control step ofcontrolling reception of the image retrieved correspondingly to theimage ID.

A 12^(th) communication apparatus of the present invention ischaracterized by comprising: an image ID receiving means for receivingan image ID transmitted from other communication apparatus; a storingmeans for storing an image comprised of at least one image of aforeground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object separatedfrom image data that is obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values determined for every pixels inaccordance with an amount of light forming an image integrated overtime; a retrieving means for retrieving the image corresponding to theimage ID received by the image ID receiving means among images stored inthe storing means; an output means for outputting the image retrieved bythe retrieving means to other communication apparatus; and a chargingmeans for performing a charge processing in accordance with the imageretrieved by the retrieving means. A 12^(th) communication method of thepresent invention is characterized by comprising: an image ID receivingstep of receiving an image ID transmitted from other communicationapparatus; a storing step of storing an image comprised of at least oneimage of a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject separated from image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time; a retrieving step of retrieving the imagecorresponding to the image ID received in the image ID receiving stepamong images stored in the storing step; an output step of outputtingthe image retrieved in the retrieving step to other communicationapparatus; and a charging step of performing a charge processing inaccordance with the image retrieved in the retrieving step.

A program of a 17^(th) recording medium of the present invention ischaracterized by comprising: an image ID reception control step ofcontrolling reception of an image ID transmitted from othercommunication apparatus; a storage control step of controlling storageof an image comprised of at least one image of a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object separated from image datathat is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; a retrievalcontrol step of controlling retrieval of the image corresponding to theimage ID received in the image ID reception control step among imagesstored in the storage control step; an output control step ofcontrolling output of the image retrieved in the retrieval control stepto other communication apparatus; and a charge control step ofcontrolling performance of a charge processing in accordance with theimage retrieved in the retrieval control step.

A 17^(th) program of the present invention is characterized by making acomputer perform: an image ID reception control step of controllingreception of an image ID transmitted from other communication apparatus;a storage control step of controlling storage of an image comprised ofat least one image of a foreground component image having foregroundobject components constituting a foreground object and a backgroundcomponent image having background object components constituting abackground object separated from image data that is obtained by an imagepickup element having a predetermined number of pixels, each having atime integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time; a retrieval control step ofcontrolling retrieval of the image corresponding to the image IDreceived in the image ID reception control step among images stored inthe storage control step; an output control step of controlling outputof the image retrieved in the retrieval control step to othercommunication apparatus; and a charge control step of controllingperformance of a charge processing in accordance with the imageretrieved in the retrieval control step.

A 13^(th) communication apparatus of the present invention ischaracterized by comprising: an input means for inputting an imagecomprised of at least one image of a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object separated from image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; a storing means for storingthe image input by the input means; and a paying means for performing apayment processing in accordance with the image stored in the storingmeans.

A separating means for separating the image into the foregroundcomponent image having foreground object components constituting theforeground object and the background component image having backgroundobject components constituting the background object may be furthercomprised and the input means may input the foreground component imageand the background component image separated by the separating means.

A synthesized image generating means for synthesizing in a predeterminedmixture ratio the foreground component image having foreground objectcomponents constituting the foreground object and the backgroundcomponent image having background object components constituting thebackground object separated from the image and generating a synthesizedimage may be further comprised and the input means may input thesynthesized image generated by the synthesized image generating means.

A separating means for separating the image into the foregroundcomponent image having foreground object components constituting theforeground object and the background component image having backgroundobject components constituting the background object and a synthesizedimage generating means for synthesizing in a predetermined mixture ratiothe foreground component image and the background component imageseparated by the separating means and generating a synthesized image maybe further comprised and the input means may input the synthesized imagegenerated by the synthesized image generating means.

A 13^(th) communication method of the present invention is characterizedby comprising: an input step of inputting an image comprised of at leastone image of a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject separated from image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time; a storing step of storing the image input in theinput step; and a paying step of performing a payment processing inaccordance with the image stored in the storing step.

A program of an 18^(th) recording medium of the present invention ischaracterized by comprising: an input control step of controlling inputof an image comprised of at least one image of a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object separated from image datathat is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; a storagecontrol step of controlling storage of the image input in the inputcontrol step; and a payment control step of controlling performance of apayment processing in accordance with the image stored in the storagecontrol step.

A 18^(th) program of the present invention is characterized by making acomputer perform: an input control step of controlling input of an imagecomprised of at least one image of a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object separated from image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; a storage control step ofcontrolling storage of the image input in the input control step; and apayment control step of controlling performance of a payment processingin accordance with the image stored in the storage control step.

A 6^(th) communication system of the present invention is characterizedin that a first communication apparatus comprises: an input means forinputting an image comprised of at least one image of a foregroundcomponent image having foreground object components constituting aforeground object and a background component image having backgroundobject components constituting a background object separated from imagedata that is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; and an imagetransmitting means for transmitting the image input by the input meansto a second communication apparatus, and the second communicationapparatus comprises: an image receiving means for receiving the imagetransmitted from the first communication apparatus; a storing means forstoring the image received by the image receiving means; and a payingmeans for performing a payment processing in accordance with the imagestored in the storing means.

The first communication apparatus may further comprise a separatingmeans for separating the image into the foreground component imagehaving foreground object components constituting the foreground objectand the background component image having background object componentsconstituting the background object and the input means may input theforeground component image and the background component image separatedby the separating means.

The first communication apparatus may further comprise a synthesizedimage generating means for synthesizing in a predetermined mixture ratiothe foreground component image having foreground object componentsconstituting the foreground object and the background component imagehaving background object components constituting the background objectseparated from the image and generating a synthesized image and theinput means may input the synthesized image generated by the synthesizedimage generating means.

The first communication apparatus may further comprise a separatingmeans for separating the image into the foreground component imagehaving foreground object components constituting the foreground objectand the background component image having background object componentsconstituting the background object and a synthesized image generatingmeans for synthesizing in a predetermined mixture ratio the foregroundcomponent image and the background component image separated by theseparating means and generating a synthesized image and the input meansmay input the synthesized image generated by the synthesized imagegenerating means.

A communication method of the 6^(th) communication system ischaracterized in that a communication method of the first communicationapparatus comprises: an input step of inputting an image comprised of atleast one image of a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject separated from image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time; and an image transmitting step of transmitting theimage input in the input step to a second communication apparatus, and acommunication method of the second communication apparatus comprises: animage receiving step of receiving the image transmitted from the firstcommunication apparatus; a storing step of storing the image received inthe image receiving step; and a paying step of performing a paymentprocessing in accordance with the image stored in the storing step. Aprogram of the 19^(th) recording medium of the present invention ischaracterized in that a program for controlling the first communicationapparatus comprises: an input control step of controlling input of animage comprised of at least one image of a foreground component imagehaving foreground object components constituting a foreground object anda background component image having background object componentsconstituting a background object separated from image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; and an image transmissioncontrol step of controlling transmission of the image input in the inputcontrol step to a second communication apparatus, and a program forcontrolling the second communication apparatus comprises: an imagereception control step of controlling reception of the image transmittedfrom the first communication apparatus; a storage control step ofcontrolling storage of the image received in the image reception controlstep; and a payment control step of controlling performance of a paymentprocessing in accordance with the image stored in the storage controlstep.

A 19^(th) program of the present invention is characterized by making acomputer for controlling a first communication apparatus perform: aninput control step of controlling input of an image comprised of atleast one image of a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject separated from image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time; and an image transmission control step ofcontrolling transmission of the image input in the input control step toa second communication apparatus, and by making a computer forcontrolling the second communication apparatus perform: an imagereception control step of controlling reception of the image transmittedfrom the first communication apparatus; a storage control step ofcontrolling storage of the image received in the image reception controlstep; and a payment control step of controlling performance of a paymentprocessing in accordance with the image stored in the storage controlstep.

A 14^(th) communication apparatus of the present invention ischaracterized by comprising: an input means for inputting an imagecomprised of at least one image of a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object separated from image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; and an image transmittingmeans for transmitting the image input by the input means to othercommunication apparatus.

A 14^(th) communication method of the present invention is characterizedby comprising: an input step of inputting an image comprised of at leastone image of a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject separated from image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time; and an image transmitting step of transmitting theimage input in the input step to other communication apparatus.

A program of a 20^(th) recording medium of the present invention ischaracterized by comprising: an input control step of controlling inputof an image comprised of at least one image of a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object separated from image datathat is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; and an imagetransmission control step of controlling transmission of the image inputin the input control step to other communication apparatus.

A 20^(th) program of the present invention is characterized by making acomputer perform: an input control step of controlling input of an imagecomprised of at least one image of a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object separated from image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; and an image transmissioncontrol step of controlling transmission of the image input in the inputcontrol step to other communication apparatus.

A 15^(th) communication apparatus of the present invention ischaracterized by comprising: an image receiving means for receiving animage transmitted from other communication apparatus; a storing meansfor storing the image received by the image receiving means; and apaying means for performing a payment processing in accordance with theimage stored in the storing means.

A 15^(th) communication method of the present invention is characterizedby comprising: an image receiving step of receiving an image transmittedfrom other communication apparatus; a storing step of storing the imagereceived in the image receiving step; and a paying step of performing apayment processing in accordance with the image stored in the storingstep.

A program of a 21^(st) recording medium of the present invention ischaracterized by comprising: an image reception control step ofcontrolling reception of an image transmitted from other communicationapparatus; a storage control step of controlling storage of the imagereceived in the image reception control step; and a payment control stepof controlling performance of a payment processing in accordance withthe image stored in the storage control step.

A 21^(st) program of the present invention is characterized by making acomputer perform: an image reception control step of controllingreception of an image transmitted from other communication apparatus; astorage control step of controlling storage of the image received in theimage reception control step; and a payment control step of controllingperformance of a payment processing in accordance with the image storedin the storage control step.

A 16^(th) communication apparatus of the present invention ischaracterized by comprising: a retrieval request information input meansfor inputting retrieval request information of a user; a storing meansfor storing an image comprised of at least one image of a foregroundcomponent image having foreground object components constituting aforeground object and a background component image having backgroundobject components constituting a background object separated from imagedata that is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; a retrievingmeans for retrieving the image corresponding to the retrieval requestinformation input by the retrieval request information input means amongimages stored in the storing means; an output means for outputting aretrieval result of the retrieving means; and a charging means forperforming a charge processing in accordance with the retrieval result.

The storing means may comprise a plurality of image storage units,respectively, provided with an image pickup element for picking up imagefor updating image picked up by the image pickup element at apredetermined time interval to store them and being capable ofcommunication by a wire or wirelessly.

The retrieval request information may include information specifying aforeground object, the storing means may further comprise a database forindicating relationship between an ID discerning the image pickupelements provided in the plurality of image storage units and an imageobject stored in the corresponding image storage unit and the retrievingmeans may retrieve the image corresponding to the information specifyingthe foreground object input by the retrieval request information inputmeans among images stored in the storing means with reference to thedatabase.

A separating means for separating image data of the image into theforeground component image having foreground object componentsconstituting the foreground object and the background component imagehaving background object components constituting the background objectmay be further comprised and the storing means may store the foregroundcomponent image and the background component image separated by theseparating means.

A synthesized image generating means for synthesizing in a predeterminedmixture ratio the foreground component image having foreground objectcomponents constituting the foreground object and the backgroundcomponent image having background object components constituting thebackground object separated from the image data of the image andgenerating a synthesized image may be further comprised and the storingmeans may store the synthesized image generated by the synthesized imagegenerating means.

A separating means for separating image data of the image into theforeground component image having foreground object componentsconstituting the foreground object and the background component imagehaving background object components constituting the background objectand a synthesized image generating means for synthesizing in apredetermined mixture ratio the foreground component image and thebackground component image separated by the separating means andgenerating a synthesized image may be further comprised and the storingmeans may store the synthesized image generated by the synthesized imagegenerating means.

The output means may output existence and nonexistence of the imagecorresponding to the retrieval request information or the retrievedimage along with existence and nonexistence of the image correspondingto the retrieval request information as the retrieval result, and thecharging means may perform the charge processing on the basis of theretrieval result output by the output means.

The charging means may generate charge information including a user ID,a communication apparatus ID and cost information corresponding to theretrieval request information in accordance with the retrieval requestinformation.

The charging means may perform the charge processing based on theretrieval result output by the output means with respect to a financialaccount of the user on the basis of the charge information.

The charging means may perform the charge processing by subtracting thenumber of points corresponding to the cost information from the numberof points of each user used for the charge processing.

The output means may output the retrieval result to a communicationapparatus of the user having finished the charge processing after thecharge processing has been finished by the charging means.

A 16^(th) communication method of the present invention is characterizedby comprising: a retrieval request information input step of inputtingretrieval request information of a user; a storing step of storing animage comprised of at least one image of a foreground component imagehaving foreground object components constituting a foreground object anda background component image having background object componentsconstituting a background object separated from image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; a retrieving step ofretrieving image corresponding to the retrieval request informationinput in the retrieval request information input step among imagesstored in the storing step; an output step of outputting a retrievalresult of the retrieving step; and a charging step of performing acharge processing in accordance with the retrieval result.

A program of a 22^(nd) recording medium of the present invention ischaracterized by comprising: a retrieval request information inputcontrol step of controlling input of a retrieval request information ofa user; a storage control step of controlling storage of an imagecomprised of at least one image of a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object separated from image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; a retrieval control step ofcontrolling retrieval of the image corresponding to the retrievalrequest information input in the retrieval request information inputcontrol step among images stored in the storage control step; an outputcontrol step of controlling output of a retrieval result of theretrieval control step; and a charge control step of controllingperformance of a charge processing in accordance with the retrievalresult.

A 22^(nd) program of the present invention is characterized by making acomputer perform: a retrieval request information input control step ofcontrolling input of a retrieval request information of a user; astorage control step of controlling storage of an image comprised of atleast one image of a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject separated from image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time; a retrieval control step of controlling retrievalof image corresponding to the retrieval request information input in theretrieval request information input control step among images stored inthe storage control step; an output control step of controlling outputof a retrieval result of the retrieval control step; and a chargecontrol step of controlling performance of a charge processing inaccordance with the retrieval result.

A 7^(th) communication system of the present invention is characterizedin that a first communication apparatus comprises: a retrieval requestinformation input means for inputting a retrieval request information ofa user; a retrieval request information transmitting means fortransmitting the retrieval request information to a second communicationapparatus; and a retrieval result receiving means for receiving aretrieval result transmitted from the second communication apparatus,and the second communication apparatus comprises: a retrieval requestinformation receiving means for receiving the retrieval requestinformation transmitted from the first communication apparatus; astoring means for storing an image comprised of at least one image of aforeground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object separatedfrom image data that is obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values determined for every pixels inaccordance with an amount of light forming an image integrated overtime; a retrieving means for retrieving the image corresponding to theretrieval request information received by the retrieval requestinformation receiving means among images stored in the storing means; anretrieval result transmitting means for transmitting the retrievalresult of the retrieving means to the first communication apparatus; anda charging means for performing a charge processing in accordance withthe retrieval result.

The storing means may comprise a plurality of image storage units,respectively, provided with an image pickup element for picking up imagefor updating image picked up by the image pickup element at apredetermined time interval to store them and being capable ofcommunication by a wire or wirelessly.

The retrieval request information may include information specifying aforeground object, the storing means may further comprise a database forindicating relationship between an ID discerning the image pickupelements provided in the plurality of image storage units and an imageobject stored in the corresponding image storage unit and the retrievingmeans may retrieve the image corresponding to the information specifyingthe foreground object input by the retrieval request information inputmeans among images stored in the storing means with reference to thedatabase.

The second communication apparatus may further comprise a separatingmeans for separating image data of the image into the foregroundcomponent image having foreground object components constituting theforeground object and the background component image having backgroundobject components constituting the background object and the storingmeans may store the foreground component image and the backgroundcomponent image separated by the separating means.

The second communication apparatus may further comprise a synthesizedimage generating means for synthesizing in a predetermined mixture ratiothe foreground component image having foreground object componentsconstituting the foreground object and the background component imagehaving background object components constituting the background objectseparated from image data of the image and generating a synthesizedimage and the storing means may store the synthesized image generated bythe synthesized image generating means.

The second communication apparatus may further comprise a separatingmeans for separating image data of the image into the foregroundcomponent image having foreground object components constituting theforeground object and the background component image having backgroundobject components constituting the background object and a synthesizedimage generating means for synthesizing in a predetermined mixture ratiothe foreground component image and the background component imageseparated by the separating means and generating a synthesized image andthe storing means may store the synthesized image generated by thesynthesized image generating means.

The output means may output existence and nonexistence of the imagecorresponding to the retrieval request information or the retrievedimage along with existence and nonexistence of the image correspondingto the retrieval request information as the retrieval result and thecharging means may perform the charge processing on the basis of theretrieval result output by the output means.

The charging means may generate charge information including a user ID,a communication apparatus ID and cost information corresponding to theretrieval request information in accordance with the retrieval requestinformation.

The charging means may perform the charge processing based on theretrieval result output by the output means with respect to a financialaccount of the user on the basis of the charge information.

The charging means may perform the charge processing by subtracting thenumber of points corresponding to the cost information from the numberof points of each user used for the charge processing.

The output means may output the retrieval result to a communicationapparatus of the user having finished the charge processing after thecharge processing has been finished by the charging means.

A communication method of the 7^(th) communication system of the presentinvention is characterized in that a communication method of the firstcommunication apparatus comprises: a retrieval request information inputstep of inputting retrieval request information of a user; a retrievalrequest information transmitting step of transmitting the retrievalrequest information to a second communication apparatus; and a retrievalresult receiving step of receiving a retrieval result transmitted fromthe second communication apparatus, and a communication method of thesecond communication apparatus comprises: a retrieval requestinformation receiving step of receiving the retrieval requestinformation transmitted from the first communication apparatus; astoring step of storing an image comprised of at least one image of aforeground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object separatedfrom image data that is obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values determined for every pixels inaccordance with an amount of light forming an image integrated overtime; a retrieving step of retrieving the image corresponding to theretrieval request information received in the retrieval requestinformation receiving step among images stored in the storing step; anretrieval result transmitting step of transmitting the retrieval resultof the retrieving step to the first communication apparatus; and acharging step of performing a charge processing in accordance with theretrieval result.

A program of a 23^(rd) recording medium of the present invention ischaracterized in that a program for controlling the first communicationapparatus comprises: a retrieval request information input control stepof controlling input of retrieval request information of a user; aretrieval request information transmission control step of controllingtransmission of the retrieval request information to a secondcommunication apparatus; and a retrieval result reception control stepof controlling reception of a retrieval result transmitted from thesecond communication apparatus, and a program for controlling the secondcommunication apparatus comprises: a retrieval request informationreception control step of controlling reception of the retrieval requestinformation transmitted from the first communication apparatus; astorage control step of controlling storage of an image comprised of atleast one image of a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject separated from image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time; a retrieval control step of controlling retrievalof the image corresponding to the retrieval request information receivedin the retrieval request information reception control step among imagesstored in the storage control step; an retrieval result transmissioncontrol step of controlling transmission of the retrieval result of theretrieval control step to the first communication apparatus; and acharge control step of controlling performance of a charge processing inaccordance with the retrieval result.

A 23^(rd) program of the present invention is characterized by making acomputer for controlling the first communication apparatus perform: aretrieval request information input control step of controlling input ofretrieval request information of a user; a retrieval request informationtransmission control step of controlling transmission of the retrievalrequest information to a second communication apparatus; and a retrievalresult reception control step of controlling reception of a retrievalresult transmitted from the second communication apparatus, and bymaking a computer for controlling the second communication apparatusperform: a retrieval request information reception control step ofcontrolling reception of the retrieval request information transmittedfrom the first communication apparatus; a storage control step ofcontrolling storage of an image comprised of at least one image of aforeground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object separatedfrom image data that is obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values determined for every pixels inaccordance with an amount of light forming an image integrated overtime; a retrieval control step of controlling retrieval of the imagecorresponding to the retrieval request information received in theretrieval request information reception control step among images storedin the storage control step; an retrieval result transmission controlstep of controlling transmission of the retrieval result of theretrieval control step to the first communication apparatus; and acharge control step of controlling performance of a charge processing inaccordance with the retrieval result.

A 17^(th) communication apparatus of the present invention ischaracterized by comprising: a retrieval request information input meansfor inputting retrieval request information of a user; a retrievalrequest information transmitting means for transmitting the retrievalrequest information to other communication apparatus; and a retrievalresult receiving means for receiving a retrieval result transmitted fromthe other communication apparatus.

A 17^(th) communication method of the present invention is characterizedby comprising: a retrieval request information input step of inputtingretrieval request information of a user; a retrieval request informationtransmitting step of transmitting the retrieval request information toother communication apparatus; and a retrieval result receiving step ofreceiving a retrieval result transmitted from the other communicationapparatus.

A program of a 24^(th) recording medium of the present invention ischaracterized by comprising: a retrieval request information inputcontrol step of controlling input of retrieval request information of auser; a retrieval request information transmission control step ofcontrolling transmission of the retrieval request information to othercommunication apparatus; and a retrieval result reception control stepof controlling reception of a retrieval result transmitted from theother communication apparatus.

A 24^(th) program of the present invention is characterized by making acomputer perform: a retrieval request information input control step ofcontrolling input of retrieval request information of a user; aretrieval request information transmission control step of controllingtransmission of the retrieval request information to other communicationapparatus; and a retrieval result reception control step of controllingreception of a retrieval result transmitted from the other communicationapparatus.

An 18^(th) communication apparatus of the present invention ischaracterized by comprising: a retrieval request information receivingmeans for receiving retrieval request information transmitted from othercommunication apparatus; a storing means for storing an image comprisedof at least one image of a foreground component image having foregroundobject components constituting a foreground object and a backgroundcomponent image having background object components constituting abackground object separated from image data that is obtained by an imagepickup element having a predetermined number of pixels, each having atime integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time; a retrieving means for retrievingthe image corresponding to the retrieval request information received bythe retrieval request information receiving means among images stored inthe storing means; an retrieval result transmitting means fortransmitting a retrieval result of the retrieving means to the othercommunication apparatus; and a charging means for performing a chargeprocessing in accordance with the retrieval result.

An 18^(th) communication apparatus of the present invention ischaracterized by comprising: a retrieval request information receivingstep of receiving retrieval request information transmitted from othercommunication apparatus; a storing step of storing an image comprised ofat least one image of a foreground component image having foregroundobject components constituting a foreground object and a backgroundcomponent image having background object components constituting abackground object separated from image data that is obtained by an imagepickup element having a predetermined number of pixels, each having atime integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time; a retrieving step of retrievingthe image corresponding to the retrieval request information received inthe retrieval request information receiving step among images stored inthe storing step; an retrieval result transmitting step of transmittinga retrieval result of the retrieving step to the other communicationapparatus; and a charging step of performing a charge processing inaccordance with the retrieval result.

A program of a 25^(th) recording medium of the present invention ischaracterized by comprising: a retrieval request information receptioncontrol step of controlling reception of retrieval request informationtransmitted from other communication apparatus; a storage control stepof controlling storage of an image comprised of at least one image of aforeground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object separatedfrom image data that is obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values determined for every pixels inaccordance with an amount of light forming an image integrated overtime; a retrieval control step of controlling retrieval of the imagecorresponding to the retrieval request information received in theretrieval request information reception control step among images storedin the storage control step; an retrieval result transmission controlstep of controlling transmission of a retrieval result of the retrievalcontrol step to the other communication apparatus; and a charge controlstep of control performance of a charge processing in accordance withthe retrieval result.

A 25^(th) program of the present invention is characterized by making acomputer perform: a retrieval request information reception control stepof controlling reception of retrieval request information transmittedfrom other communication apparatus; a storage control step ofcontrolling storage of an image comprised of at least one image of aforeground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object separatedfrom image data that is obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values determined for every pixels inaccordance with an amount of light forming an image integrated overtime; a retrieval control step of controlling retrieval of the imagecorresponding to the retrieval request information received in theretrieval request information reception control step among images storedin the storage control step; an retrieval result transmission controlstep of controlling transmission of a retrieval result of the retrievalcontrol step to the other communication apparatus; and a charge controlstep of control performance of a charge processing in accordance withthe retrieval result.

A 19^(th) communication apparatus of the present invention ischaracterized by comprising: a foreground component image discerninginformation input means for inputting a foreground component imagediscerning information indicative of a foreground component image havingforeground object components constituting a foreground object of imagedata that is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; a backgroundcomponent image discerning information input means for inputting abackground component image discerning information indicative of abackground component image having background object componentsconstituting a background object of the image data; an image positionalinformation generating means for generating a foreground component imagepositional information and a background component image positionalinformation corresponding to the foreground component image discerninginformation and the background component image discerning information;and an output means for outputting the foreground component imagepositional information and the background component image positionalinformation generated by the image positional information generatingmeans.

A mixture ratio generating means for generating a mixture ratio of anarea in which the foreground component image and the backgroundcomponent image are mixed from the image data that is obtained by theimage pickup element having the predetermined number of pixels, eachhaving a time integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming the image integrated over time, a separating means forseparating the image data into the foreground component image and thebackground component image according to the mixture ratio, a discerninginformation generating means for generating the foreground componentimage discerning information based on the foreground component image andthe background component image discerning information based on thebackground component image, respectively, and a discerning informationsupplying means for supplying the foreground component image discerninginformation generated by the discerning information generating means tothe foreground component image discerning information input means andthe background component image discerning information generated by thediscerning information generating means to the background componentimage discerning information input means, respectively, may be furthercomprised.

A storing means for storing at a predetermined position the foregroundcomponent image and the background component image separated from theimage data by the separating means and a positional information storingmeans for storing each positional information of the foregroundcomponent image and the background component image of the image datastored by the storing means may be further comprised and the imagepositional information generating means may generate the foregroundcomponent image positional information and the background componentimage positional information corresponding to the foreground componentimage discerning information and the background component imagediscerning information on the basis of the positional information storedby the positional information storing means and indicative of positionsat which the foreground component image and the background componentimage are stored.

The storing means may store the foreground component image and thebackground component image separated from the image data by theseparating means in other communication apparatus connected through anetwork, the positional information storing means may store eachpositional information on the network of the foreground component imageand the background component image of the image data stored in the othercommunication apparatus connected via the network by the storing means,and the image positional information generating means may generate theforeground component image positional information and the backgroundcomponent image positional information corresponding to the foregroundcomponent image discerning information and the background componentimage discerning information on the basis of the positional informationon the network of the foreground component image and the backgroundcomponent image of the image data stored by the positional informationstoring means.

The storing means may store at a predetermined position the foregroundcomponent image and the background component image separated from theimage data by the separating means along with IDs corresponding to theforeground component image and the background component image,respectively.

A comparing means for comparing the respective IDs of the foregroundcomponent image and the background component image separated from theimage data by the separating means with IDs of the foreground componentimage or the background component image previously stored by the storingmeans may be further comprised, and the storing means may store at apredetermined position the foreground component image and the backgroundcomponent image separated from the image data by the separating meansalong with IDs corresponding to the foreground component image and thebackground component image, respectively, on the basis of the comparisonresult of the comparing means.

When the comparison results of the comparing means are matched, theimage positional information generating means may generate theforeground component image positional information and the backgroundcomponent image positional information corresponding to the foregroundcomponent image discerning information and the background componentimage discerning information on the basis of the positional informationof the foreground component image or the background component imagepreviously stored by the positional information storing means.

The foreground component image discerning information input means mayinput the foreground component image having the foreground objectcomponents constituting the foreground object of the image data as theforeground component image discerning information and the backgroundcomponent image discerning information input means may input thebackground component image having the background object componentsconstituting the background object of the image data as the backgroundcomponent image discerning information.

A charging means for performing a charge processing corresponding tooutput of the foreground component image positional information and thebackground component image positional information may be furthercomprised.

The charging means may generate charge information including a user ID,an ID of itself and cost information corresponding to output of theforeground component image positional information and the backgroundcomponent image positional information.

The charging means may perform the charge processing with respect to afinancial account of the user on the basis of the charge information.

The charging means may perform the charge processing by subtracting thenumber of points corresponding to the cost information from the numberof points of each user used for the charge processing.

The output means may output the foreground component image positionalinformation and the background component image positional informationafter the charge processing has been finished by the charging means, ina way that only the user can obtain them.

A 19^(th) communication method of the present invention is characterizedby comprising: a foreground component image discerning information inputstep of inputting a foreground component image discerning informationindicative of a foreground component image having foreground objectcomponents constituting a foreground object of image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; a background componentimage discerning information input step of inputting a backgroundcomponent image discerning information indicative of a backgroundcomponent image having background object components constituting abackground object of the image data; an image positional informationgenerating step of generating a foreground component image positionalinformation and a background component image positional informationcorresponding to the foreground component image discerning informationand the background component image discerning information; and an outputstep of outputting the foreground component image positional informationand the background component image positional information generated inthe image positional information generating step.

A program of a 26^(th) recording medium of the present invention ischaracterized by comprising: a foreground component image discerninginformation input control step of controlling input of a foregroundcomponent image discerning information indicative of a foregroundcomponent image having foreground object components constituting aforeground object of image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time; a background component image discerninginformation input control step of controlling input of a backgroundcomponent image discerning information indicative of a backgroundcomponent image having background object components constituting abackground object of the image data; an image positional informationgeneration control step of controlling generation of a foregroundcomponent image positional information and a background component imagepositional information corresponding to the foreground component imagediscerning information and the background component image discerninginformation; and an output control step of controlling output of theforeground component image positional information and the backgroundcomponent image positional information generated in the image positionalinformation generation control step.

A 26^(th) program of the present invention is characterized by making acomputer perform: a foreground component image discerning informationinput control step of controlling input of a foreground component imagediscerning information indicative of a foreground component image havingforeground object components constituting a foreground object of imagedata that is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; a backgroundcomponent image discerning information input control step of controllinginput of a background component image discerning information indicativeof a background component image having background object componentsconstituting a background object of the image data; an image positionalinformation generation control step of controlling generation of aforeground component image positional information and a backgroundcomponent image positional information corresponding to the foregroundcomponent image discerning information and the background componentimage discerning information; and an output control step of controllingoutput of the foreground component image positional information and thebackground component image positional information generated in the imagepositional information generation control step.

A 8^(th) communication system of the present invention is characterizedin that a first communication apparatus comprises: a foregroundcomponent image discerning information input means for inputting aforeground component image discerning information indicative of aforeground component image having foreground object componentsconstituting a foreground object of image data that is obtained by animage pickup element having a predetermined number of pixels, eachhaving a time integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time; a background component imagediscerning information input means for inputting a background componentimage discerning information indicative of a background component imagehaving background object components constituting a background object ofthe image data; a first transmitting means for transmitting theforeground component image discerning information and the backgroundcomponent image discerning information to a second communicationapparatus; and a positional information receiving means for receiving aforeground component image positional information and a backgroundcomponent image positional information transmitted from the secondcommunication apparatus, and the second communication apparatuscomprises: a foreground component image discerning information receivingmeans for receiving the foreground component image discerninginformation transmitted from the first communication apparatus; abackground component image discerning information receiving means forreceiving the background component image discerning informationtransmitted from the first communication apparatus; an image positionalinformation generating means for generating a foreground component imagepositional information and a background component image positionalinformation corresponding to the foreground component image discerninginformation and the background component image discerning information;and a second transmitting means for transmitting the foregroundcomponent image positional information and the background componentimage positional information generated by the image positionalinformation generating means to the first communication apparatus.

The second communication apparatus may further comprise: a mixture ratiogenerating means for generating a mixture ratio of an area in which theforeground component image and the background component image are mixedfrom the image data that is obtained by the image pickup element havingthe predetermined number of pixels, each having a time integrationeffect, and that is comprised of pixel values determined for everypixels in accordance with an amount of light forming the imageintegrated over time; a separating means for separating the image datainto the foreground component image and the background component imageaccording to the mixture ratio; a discerning information generatingmeans for generating the foreground component image discerninginformation on the basis of the foreground component image and thebackground component image discerning information on the basis of thebackground component image, respectively; and a discerning informationsupplying means for supplying the foreground component image discerninginformation generated by the discerning information generating means tothe foreground, component image discerning information input means andthe background component image discerning information generated by thediscerning information generating means to the background componentimage discerning information input means, respectively.

A storing means for storing at a predetermined position the foregroundcomponent image and the background component image separated from theimage data by the separating means and a positional information storingmeans for storing each positional information of the foregroundcomponent image and the background component image of the image datastored by the storing means may be further comprised, and the imagepositional information generating means may generate the foregroundcomponent image positional information and the background componentimage positional information corresponding to the foreground componentimage discerning information and the background component imagediscerning information on the basis of the positional information storedby the positional information storing means and indicative of positionsat which the foreground component image and the background componentimage are stored.

The storing means may store the foreground component image and thebackground component image separated from the image data by theseparating means in other communication apparatus connected through anetwork, the positional information storing means may store eachpositional information on the network of the foreground component imageand the background component image of the image data stored in the othercommunication apparatus connected via the network by the storing means,and the image positional information generating means may generate theforeground component image positional information and the backgroundcomponent image positional information corresponding to the foregroundcomponent image discerning information and the background componentimage discerning information on the basis of each positional informationon the network of the foreground component image and the backgroundcomponent image of the image data stored by the positional informationstoring means.

The storing means may store at a predetermined position the foregroundcomponent image and the background component image separated from theimage data by the separating means along with IDs corresponding to theforeground component image and the background component image,respectively.

The second communication apparatus may further comprise a comparingmeans for comparing the respective IDs of the foreground component imageand the background component image separated from the image data by theseparating means with IDs of the foreground component image or thebackground component image previously stored by the storing means, andthe storing means may store at a predetermined position the foregroundcomponent image and the background component image separated from theimage data by the separating means along with IDs corresponding to theforeground component image and the background component image,respectively, on the basis of the comparison result of the comparingmeans.

When the comparison results of the comparing means are matched, theimage positional information generating means may generate theforeground component image positional information and the backgroundcomponent image positional information corresponding to the foregroundcomponent image discerning information and the background componentimage discerning information on the basis of the positional informationof the foreground component image or the background component imagepreviously stored by the positional information storing means.

The foreground component image discerning information input means mayinput the foreground component image having the foreground objectcomponents constituting the foreground object of the image data as theforeground component image discerning information and the backgroundcomponent image discerning information input means may input thebackground component image having the background object componentsconstituting the background object of the image data as the backgroundcomponent image discerning information.

The second communication apparatus may further comprise a charging meansfor performing a charge processing corresponding to output of theforeground component image positional information and the backgroundcomponent image positional information.

The charging means may generate charge information including a user ID,an ID of the second communication apparatus and cost informationcorresponding to output of the foreground component image positionalinformation and the background component image positional information.

The charging means may perform the charge processing with respect to afinancial account of the user on the basis of the charge information.

The charging means may perform the charge processing by subtracting thenumber of points corresponding to the cost information from the numberof points of each user used for the charge processing.

The output means may output the foreground component image positionalinformation and the background component image positional information tothe user having finished the charge processing after the chargeprocessing has been finished by the charging means.

A communication method of the 8^(th) communication system of the presentinvention is characterized in that a communication method of the firstcommunication apparatus comprises: a foreground component imagediscerning information input step of inputting a foreground componentimage discerning information indicative of a foreground component imagehaving foreground object components constituting a foreground object ofimage data that is obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values determined for every pixels inaccordance with an amount of light forming an image integrated overtime; a background component image discerning information input step ofinputting a background component image discerning information indicativeof a background component image having background object componentsconstituting a background object of the image data; a first transmittingstep of transmitting the foreground component image discerninginformation and the background component image discerning information toa second communication apparatus; and a positional information receivingstep of receiving a foreground component image positional informationand a background component image positional information transmitted fromthe second communication apparatus, and a communication method of thesecond communication apparatus comprises: a foreground component imagediscerning information receiving step of receiving the foregroundcomponent image discerning information transmitted from the firstcommunication apparatus; a background component image discerninginformation receiving step of receiving the background component imagediscerning information transmitted from the first communicationapparatus; an image positional information generating step of generatingthe foreground component image positional information and the backgroundcomponent image positional information corresponding to the foregroundcomponent image discerning information and the background componentimage discerning information; and a second transmitting step oftransmitting the foreground component image positional information andthe background component image positional information generated in theimage positional information generating step to the first communicationsystem.

A program of a 27^(th) recording medium of the present invention ischaracterized in that a program for controlling the first communicationapparatus comprises: a foreground component image discerning informationinput control step of controlling input of a foreground component imagediscerning information indicative of a foreground component image havingforeground object components constituting a foreground object of imagedata that is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; a backgroundcomponent image discerning information input control step of controllinginput of a background component image discerning information indicativeof a background component image having background object componentsconstituting a background object of the image data; a first transmissioncontrol step of controlling transmission of the foreground componentimage discerning information and the background component imagediscerning information to a second communication apparatus; and apositional information reception control step of controlling receptionof a foreground component image positional information and a backgroundcomponent image positional information transmitted from the secondcommunication apparatus, and a program for controlling the secondcommunication apparatus comprises: a foreground component imagediscerning information reception control step of controlling receptionof the foreground component image discerning information transmittedfrom the first communication apparatus; a background component imagediscerning information reception control step of controlling receptionof the background component image discerning information transmittedfrom the first communication apparatus; an image positional informationgeneration control step of controlling generation of the foregroundcomponent image positional information and the background componentimage positional information corresponding to the foreground componentimage discerning information and the background component imagediscerning information; and a second transmission control step ofcontrolling transmission of the foreground component image positionalinformation and the background component image positional informationgenerated in the image positional information generation control step tothe first communication system.

A 27^(th) program of the present invention is characterized by making acomputer for controlling a first communication apparatus perform: aforeground component image discerning information input control step ofcontrolling input of a foreground component image discerning informationindicative of a foreground component image having foreground objectcomponents constituting a foreground object of image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; a background componentimage discerning information input control step of controlling input ofa background component image discerning information indicative of abackground component image having background object componentsconstituting a background object of the image data; a first transmissioncontrol step of controlling transmission of the foreground componentimage discerning information and the background component imagediscerning information to a second communication apparatus; and apositional information reception control step of controlling receptionof a foreground component image positional information and a backgroundcomponent image positional information transmitted from the secondcommunication apparatus, and by making a computer for controlling thesecond communication apparatus perform: a foreground component imagediscerning information reception control step of controlling receptionof the foreground component image discerning information transmittedfrom the first communication apparatus; a background component imagediscerning information reception control step of controlling receptionof the background component image discerning information transmittedfrom the first communication apparatus; an image positional informationgeneration control step of controlling generation of the foregroundcomponent image positional information and the background componentimage positional information corresponding to the foreground componentimage discerning information and the background component imagediscerning information; and a second transmission control step ofcontrolling transmission of the foreground component image positionalinformation and the background component image positional informationgenerated in the image positional information generation control step tothe first communication system.

A 20^(th) communication apparatus of the present invention ischaracterized by comprising: a foreground component image discerninginformation input means for inputting a foreground component imagediscerning information indicative of a foreground component image havingforeground object components constituting a foreground object of imagedata that is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; a backgroundcomponent image discerning information input means for inputting abackground component image discerning information indicative of abackground component image having background object componentsconstituting a background object of the image data; a transmitting meansfor transmitting the foreground component image discerning informationand the background component image discerning information to othercommunication apparatus through a network; and a positional informationreceiving means for receiving a foreground component image positionalinformation and a background component image positional informationtransmitted from the other communication apparatus through a network.

A 20^(th) communication method of the present invention is characterizedby comprising: a foreground component image discerning information inputstep of inputting a foreground component image discerning informationindicative of a foreground component image having foreground objectcomponents constituting a foreground object of image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; a background componentimage discerning information input step of inputting a backgroundcomponent image discerning information indicative of a backgroundcomponent image having background object components constituting abackground object of the image data; a transmitting step of transmittingthe foreground component image discerning information and the backgroundcomponent image discerning information to other communication apparatusthrough a network; and a positional information receiving step ofreceiving a foreground component image positional information and abackground component image positional information transmitted from theother communication apparatus via the network.

A program of a 28^(th) recording medium of the present invention ischaracterized by comprising: a foreground component image discerninginformation input control step of controlling input of a foregroundcomponent image discerning information indicative of a foregroundcomponent image having foreground object components constituting aforeground object of image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time; a background component image discerninginformation input control step of controlling input of a backgroundcomponent image discerning information indicative of a backgroundcomponent image having background object components constituting abackground object of the image data; a transmission control step ofcontrolling transmission of the foreground component image discerninginformation and the background component image discerning information toother communication apparatus through a network; and a positionalinformation reception control step of controlling reception of aforeground component image positional information and a backgroundcomponent image positional information transmitted from the othercommunication apparatus via the network.

A 28^(th) program of the present invention is characterized by making acomputer perform: a foreground component image discerning informationinput control step of controlling input of a foreground component imagediscerning information indicative of a foreground component image havingforeground object components constituting a foreground object of imagedata that is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; a backgroundcomponent image discerning information input control step of controllinginput of a background component image discerning information indicativeof a background component image having background object componentsconstituting a background object of the image data; a transmissioncontrol step of controlling transmission of the foreground componentimage discerning information and the background component imagediscerning information to other communication apparatus through anetwork; and a positional information reception control step ofcontrolling reception of a foreground component image positionalinformation and a background component image positional informationtransmitted from the other communication apparatus via the network.

A 21^(st) communication apparatus of the present invention ischaracterized by comprising: a foreground component image discerninginformation receiving means for receiving the foreground component imagediscerning information of image data transmitted from othercommunication apparatus through a network, the image data being obtainedby an image pickup element having a predetermined number of pixels, eachhaving a time integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time; a background component imagediscerning information receiving means for receiving the backgroundcomponent image discerning information transmitted from the othercommunication apparatus; an image positional information generatingmeans for generating a foreground component image positional informationand a background component image positional information corresponding tothe foreground component image discerning information and the backgroundcomponent image discerning information; and a transmitting means fortransmitting the foreground component image positional information andthe background component image positional information generated by theimage positional information generating means to the other communicationapparatus via the network.

A 21^(st) communication method of the present invention is characterizedby comprising: a foreground component image discerning informationreceiving step of receiving the foreground component image discerninginformation of image data transmitted from other communication apparatusthrough a network, the image data being obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time; a background component image discerninginformation receiving step of receiving the background component imagediscerning information transmitted from the other communicationapparatus; an image positional information generating step of generatinga foreground component image positional information and a backgroundcomponent image positional information corresponding to the foregroundcomponent image discerning information and the background componentimage discerning information; and a transmitting step of transmittingthe foreground component image positional information and the backgroundcomponent image positional information generated in the image positionalinformation generating step to the other communication apparatus via thenetwork.

A program of a 29^(th) recording medium of the present invention ischaracterized by comprising: a foreground component image discerninginformation reception control step of controlling reception of theforeground component image discerning information of image datatransmitted from other communication apparatus through a network, theimage data being obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values determined for every pixels inaccordance with an amount of light, forming an image integrated overtime; a background component image discerning information receptioncontrol step of controlling reception of the background component imagediscerning information transmitted from the other communicationapparatus; an image positional information generation control step ofcontrolling generation of a foreground component image positionalinformation and a background component image positional informationcorresponding to the foreground component image discerning informationand the background component image discerning information; and atransmission control step of controlling transmission of the foregroundcomponent image positional information and the background componentimage positional information generated in the image positionalinformation generation control step to the other communication systemvia the network.

A 29^(th) program of the present invention is characterized by making acomputer perform: a foreground component image discerning informationreception control step of controlling reception of the foregroundcomponent image discerning information of image data transmitted fromother communication apparatus through a network, the image data beingobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; a background componentimage discerning information reception control step of controllingreception of the background component image discerning informationtransmitted from the other communication apparatus; an image positionalinformation generation control step of controlling generation of aforeground component image positional information and a backgroundcomponent image positional information corresponding to the foregroundcomponent image discerning information and the background componentimage discerning information; and a transmission control step ofcontrolling transmission of the foreground component image positionalinformation and the background component image positional informationgenerated in the image positional information generation control step tothe other communication system via the network.

In the 1^(st) communication system and method and the 2^(nd) program ofthe present invention, by a first communication apparatus, requestedinformation of a user is input from among a plurality of encoded datawhich are generated on the basis of predetermined image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; the input requestinformation is transmitted to the second communication apparatus; andthe encoded data generated on the basis of the image data correspondingto the requested information transmitted from the second communicationapparatus is received, and by the second communication apparatus, therequested information transmitted from the first communication apparatusis received; the image data is separated into a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object; and the encoded datagenerated when the image is separated into the foreground componentimage and the background component image on the basis of the receivedrequest information is transmitted to the first communication apparatus.

In the 2^(nd) communication apparatus and method and the 3^(rd) programof the present invention, the request requested information of a user isinput from among a plurality of encoded data which are generated on thebasis of predetermined image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time; the input request information is transmitted toother communication apparatus; and the encoded data generated on thebasis of the image data corresponding to the requested information,transmitted from other communication apparatus, is received.

In the 3^(rd) communication apparatus and method and the 4^(th) programof the present invention, the requested information transmitted fromother communication apparatus is received; the image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time is separated into aforeground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object; and theencoded data generated when the image is separated into the foregroundcomponent image and the background component image on the basis of thereceived request information is transmitted to other communicationapparatus.

In the 2^(nd) communication system and method and the 5^(th) program ofthe present invention, by the first communication apparatus, the requestrequested information of a user is input from among a plurality ofencoded data which are generated on the basis of predetermined imagedata that is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time; the inputrequest information is transmitted to one of the second communicationapparatus to the fourth communication apparatus; and the encoded datagenerated on the basis of the image data corresponding to the requestedinformation, transmitted from the one of the second communicationapparatus to the fourth communication apparatus, is received, by thesecond communication apparatus, the requested information transmittedfrom the first communication apparatus is received; the image data isseparated into a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject; and the foreground component image and the background componentimage separated from the image on the basis of the received requestinformation are transmitted as the encoded data to the firstcommunication apparatus, by the third communication apparatus, therequested information transmitted from the first communication apparatusis received; area information discerning any of a foreground area havingforeground object components constituting a foreground object of theimage data, a background area having background object componentsconstituting a background object of the predetermined image data and amixed area in which the foreground area and the background area aremixed is generated; and the generated area information based on thereceived request information is transmitted as the encoded data to thefirst communication apparatus, and by the fourth communicationapparatus, the requested information transmitted from the firstcommunication apparatus is received; a mixture ratio indicating a ratioin which the foreground area and the background area are mixed in themixed area of the image data is generated; and the mixture ratiogenerated on the basis of the received request information istransmitted to the first communication apparatus.

In the 4^(th) communication apparatus and method and the 6^(th) programof the present invention, the requested information of a user is input;in accordance with the requested information, a foreground componentimage having foreground object components constituting a foregroundobject and a background component image having background objectcomponents constituting a background object of the image data of apredetermined image data that is obtained by an image pickup elementhaving a predetermined number of pixels, each having a time integrationeffect, and that is comprised of pixel values determined for everypixels in accordance with an amount of light forming an image integratedover time are synthesized and a synthesized image is generated; and thegenerated synthesized image is output.

In the 3^(rd) communication system and method and the 7^(th) program ofthe present invention, by the first communication apparatus, therequested information of a user is input; the input request informationis transmitted to the second communication apparatus; and in accordancewith the requested information, the synthesized image transmitted fromthe second communication apparatus is input, and by the secondcommunication apparatus, the requested information transmitted from thefirst communication apparatus is received; in accordance with therequested information, a foreground component image having foregroundobject components constituting a foreground object and a backgroundcomponent image having background object components constituting abackground object of predetermined image data that is obtained by animage pickup element having a predetermined number of pixels, eachhaving a time integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time are synthesized and a synthesizedimage is generated; and the generated synthesized image is transmittedto the first communication apparatus.

In the 5^(th) communication apparatus and method and the 8^(th) programof the present invention, the requested information of a user is input;the input request information is transmitted to other communicationapparatus; and the synthesized data transmitted from other communicationapparatus in accordance with the requested information is received.

In the 6^(th) communication apparatus and method and the 9^(th) programof the present invention, the requested information transmitted fromother communication apparatus is received; in accordance with therequested information, a foreground component image having foregroundobject components constituting a foreground object and a backgroundcomponent image having background object components constituting abackground object of predetermined image data that is obtained by animage pickup element having a predetermined number of pixels, eachhaving a time integration effect, and that is comprised of pixel valuesdetermined for every pixels in accordance with an amount of lightforming an image integrated over time are synthesized and thesynthesized image is generated; and the generated synthesized image istransmitted to other communication apparatus.

In the 7^(th) communication apparatus and method and the 10^(th) programof the present invention, the requested information of a user is input;in accordance with the requested information, a mixture ratio indicatinga ratio in which a foreground component image having foreground objectcomponents constituting a foreground object to a background componentimage having background object components constituting a backgroundobject of the image data are mixed is estimated, the image data being isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time; on the basis of the mixtureratio, the image data is separated into the foreground component imageand the background component image; the separated foreground componentimage and any background component image, or the separated backgroundcomponent image and any foreground component image are synthesized in apredetermined mixture ratio and a synthesized image is generated; andthe synthesized image is output.

In the 4^(th) communication system and method and the 11^(th) program ofthe present invention, by the first communication apparatus, therequested information of a user is input; the input request informationis transmitted to the second transmitting apparatus; and the synthesizedimage transmitted from the second communication apparatus is received,and by the second communication apparatus, the requested informationtransmitted from the first communication apparatus is received; amixture ratio indicating a ratio in which a foreground component imagehaving foreground object components constituting a foreground object anda background component image having background object componentsconstituting a background object of the image data are mixed isestimated on the basis of the requested information, the image databeing obtained by an image pickup element having a predetermined numberof pixels, each having a time integration effect, and that is comprisedof pixel values determined for every pixels in accordance with an amountof light forming an image integrated over time; the image data isseparated into the foreground component image and the backgroundcomponent image on the basis of the mixture ratio; the separatedforeground component image and any background component image or theseparated background component image and any foreground component imageare synthesized in a predetermined mixture ratio and the synthesizedimage is generated; and the synthesized image is transmitted to thefirst communication apparatus.

In the 8^(th) communication apparatus and method and the 12^(th) programof the present invention, the requested information of a user is input;the requested information is transmitted to other communicationapparatus; and the synthesized image transmitted from othercommunication apparatus is received.

In the 9^(th) communication apparatus and method and the 13^(th) programof the present invention, the requested information transmitted fromother communication apparatus is received; a mixture ratio indicating aratio in which a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject of the image data are mixed is estimated on the basis of therequested information, the image data being obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time; the image data is separated into the foregroundcomponent image and the background component image on the basis of themixture ratio; the separated foreground component image and anybackground component image or the separated background component imageand any foreground component image are synthesized in a predeterminedratio and the synthesized image is generated; and the synthesized imageis transmitted to other communication apparatus.

In the 10^(th) communication apparatus and method and the 14^(th)program of the present invention, an ID discerning an image requested bya user is input; an image comprised of at least one image of aforeground component image having foreground object componentsconstituting a foreground object and a background component image havingbackground object components constituting a background object separatedfrom the image data that is obtained by an image pickup element having apredetermined number of pixels, each having a time integration effect,and that is comprised of pixel values determined for every pixels inaccordance with an amount of light forming an image integrated overtime, is stored; the image corresponding to the input image ID isretrieved from among the stored images; the retrieved image istransmitted to the communication apparatus of the user; and the chargeprocessing is performed in accordance with the retrieved image.

In the 5^(th) communication system and method and the 15^(th) program ofthe present invention, by the first communication apparatus, an IDdiscerning an image requested by a user is input; the image ID istransmitted to a second communication apparatus; and the image retrievedcorrespondingly to the image ID is received, and by the secondcommunication apparatus, the image ID transmitted from the firstcommunication apparatus is received; the image comprised of at least oneimage of a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject separated from image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time, is stored; the image corresponding to the receivedimage ID is retrieved from among the stored images; the retrieved imageis output to the first communication apparatus; and a charge processingis performed in accordance with the retrieved image.

In the 11^(th) communication apparatus and method and the 16^(th)program of the present invention, an ID discerning an image requested bya user is input; the image ID is transmitted to other communicationapparatus; and the image retrieved correspondingly to the image ID isreceived.

In the 12^(th) communication apparatus and method and the 17^(th)program of the present invention, an image ID transmitted from othercommunication apparatus is received; the image comprised of at least oneimage of a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject separated from image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time, is stored; the image corresponding to the receivedimage ID is retrieved from among the stored images; the retrieved imageis output to other communication apparatus; and a charge processing isperformed in accordance with the retrieved image.

In the 13^(th) communication apparatus and method and the 18^(th)program of the present invention, the image comprised of at least oneimage of a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject separated from image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated overtime, is input; the input image is stored; and a paymentprocessing is performed in accordance with the stored image.

In the 6^(th) communication system and method and the 19^(th) program ofthe present invention, by a first communication apparatus, the imagecomprised of at least one image of a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object separated from image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time, is input; and the inputimage is transmitted to a second communication apparatus, and by thesecond communication apparatus, the image transmitted from the firstcommunication apparatus is received; the received image are stored; anda payment processing is performed in accordance with the stored image.In the 14^(th) communication apparatus and method and the 20^(th)program of the present invention, the image comprised of at least oneimage of a foreground component image having foreground objectcomponents constituting a foreground object and a background componentimage having background object components constituting a backgroundobject separated from image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect, and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time, is input; and the input image is transmitted toother communication apparatus.

In the 15^(th) communication apparatus and method and the 21^(st)program of the present invention, an image transmitted from othercommunication apparatus is received; the received image is stored; and apayment processing is performed in accordance with the stored image.

In the 16^(th) communication apparatus and method and the 22^(nd)program of the present invention, a retrieval request information of auser is input; the image comprised of at least one image of a foregroundcomponent image having foreground object components constituting aforeground object and a background component image having backgroundobject components constituting a background object separated from imagedata that is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time, is input; theimage corresponding to the input retrieval request information isretrieved from among the stored images; the retrieval result is output;and a charge processing is performed in accordance with the retrievalresult.

In the 7^(th) communication system and method and the 23^(rd) program ofthe present invention, by a first communication apparatus, a retrievalrequest information of a user is input; the retrieval requestinformation is transmitted to a second communication apparatus; and theretrieval result transmitted from the second communication apparatus isreceived, and by the second communication apparatus, the retrievalrequest information transmitted from the first communication apparatusis received; the image comprised of at least one image of a foregroundcomponent image having foreground object components constituting aforeground object and a background component image having backgroundobject components constituting a background object separated from imagedata that is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time, is stored; theimage corresponding to the received retrieval request information isretrieved from among the stored images; the retrieval result istransmitted to the first communication apparatus; and a chargeprocessing is performed in accordance with the retrieval result.

In the 17^(th) communication apparatus and method and the 24^(th)program of the present invention, retrieval request information of auser is input; the retrieval request information is transmitted to othercommunication apparatus; and the retrieval result transmitted from theother communication apparatus is received.

In the 18^(th) communication apparatus and method and the 25^(th)program of the present invention, the retrieval request informationtransmitted from other communication apparatus is received; the imagecomprised of at least one image of a foreground component image havingforeground object components constituting a foreground object and abackground component image having background object componentsconstituting a background object separated from image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time, is stored; the imagecorresponding to the received retrieval request information is retrievedfrom among the stored images; the retrieval result is transmitted to theother communication apparatus; and a charge processing is performed inaccordance with the retrieval result.

In the 19^(th) communication apparatus and method and the 26^(th)program of the present invention, foreground component image discerninginformation indicative of a foreground component image having foregroundobject components constituting a foreground object of image data that isobtained by an image pickup element having a predetermined number ofpixels, each having a time integration effect, and that is comprised ofpixel values determined for every pixels in accordance with an amount oflight forming an image integrated over time, is input; backgroundcomponent image discerning information indicative of a backgroundcomponent image having background object components constituting abackground object of the image data is input; foreground component imagepositional information and background component image positionalinformation corresponding to the foreground component image discerninginformation and the background component image discerning informationare generated; and the generated foreground component image positionalinformation and the generated background component image positionalinformation are output.

In the 8^(th) communication system and method and the 27^(th) program ofthe present invention, by a first communication apparatus, a foregroundcomponent image discerning information indicative of a foregroundcomponent image having foreground object components constituting aforeground object of image data that is obtained by an image pickupelement having a predetermined number of pixels, each having a timeintegration effect; and that is comprised of pixel values determined forevery pixels in accordance with an amount of light forming an imageintegrated over time, is input; a background component image discerninginformation indicative of a background component image having backgroundobject components constituting a background object of the image data isinput; the foreground component image discerning information and thebackground component image discerning information are transmitted to asecond communication apparatus; and a foreground component imagepositional information and a background component image positionalinformation transmitted from the second communication apparatus arereceived, and by a second communication apparatus, the foregroundcomponent image discerning information transmitted from the firstcommunication apparatus is received; the background component imagediscerning information transmitted from the first communicationapparatus is received; the foreground component image positionalinformation and the background component image positional informationcorresponding to the foreground component image discerning informationand the background component image discerning information are generated;and the generated foreground component image positional information andthe generated background component image positional information aretransmitted to the first communication apparatus.

In the 20^(th) communication apparatus and method and the 28^(th)program of the present invention, a foreground component imagediscerning information indicative of a foreground component image havingforeground object components constituting a foreground object of imagedata that is obtained by an image pickup element having a predeterminednumber of pixels, each having a time integration effect, and that iscomprised of pixel values determined for every pixels in accordance withan amount of light forming an image integrated over time, is input; abackground component image discerning information indicative of abackground component image having background object componentsconstituting a background object of the image data is input; theforeground component image discerning information and the backgroundcomponent image discerning information are transmitted to othercommunication apparatus through a network; a foreground component imagepositional information and a background component image positionalinformation transmitted from the other communication apparatus through anetwork are received.

In the 21^(st) communication apparatus and method and the 29^(th)program of the present invention, the foreground component imagediscerning information of image data transmitted from othercommunication apparatus through a network is received, the image databeing obtained by an image pickup element having a predetermined numberof pixels, each having a time integration effect, and that is comprisedof pixel values determined for every pixels in accordance with an amountof light forming an image integrated over time; the background componentimage discerning information transmitted from the other communicationapparatus is received; a foreground component image positionalinformation and a background component image positional informationcorresponding to the foreground component image discerning informationand the background component image discerning information are generated;and the generated foreground component image positional information andthe generated background component image positional information aretransmitted to the other communication apparatus via the network.

As a result, it is possible to encrypt an image by adding motion blur tothe image, and also to decrypt the encrypted image by eliminating themotion blur, that is, adding motion blur inverse to the motion bluradded in the encryption process.

Further, it is possible to obtain a clear foreground component imagehaving been eliminated with motion blur by adjusting the motion blurcontained in the foreground component image, and to generate asynthesized image appearing seemingly natural by combining the motionblur adjusted foreground component image with a background componentimage. As a result, by adjusting the motion blur in the foregroundcomponent image and by synthesizing an image by processing the resultingforeground component with a background component, it is possible tocorrect images more naturally.

Moreover, since the image obtained by separating, synthesizing orcorrecting an image on the basis of mixed state of the image, can bepurchased or sold via the network 1, the separated, synthesized orcorrected image can be effectively utilized by selling the imagesprocessed by the users.

Furthermore, it is possible to retrieve the images which are beingphotographed, or have been photographed and stored by the cameraterminal units 28-1 to 28-n by a simple input of the retrievalconditions.

Moreover, by storing images in a predetermined storage server ratherthan in user's computer, and by reading out and utilizing the images onthe basis of positional information of the storage server as needed,pre-existing image data can be called out and utilized as needed withoutmaintaining them in their own communication apparatus, such that imagescan be efficiently utilized.

According to above descriptions, it is possible to encrypt image byadding motion blur to the image and decrypt the image by eliminating themotion blur. Further, it is also possible to eliminate motion blur togenerate a clear image and correct an image more naturally to synthesizean image in which motion blur is adjusted by combining the motion bluradjusted foreground component with a background image. Further, it ispossible to effectively utilize via the network images separated,synthesized or corrected on the basis of mixed state of the image.Moreover, it is also possible to retrieve the separated, synthesized orcorrected images via the network. In addition, it is possible to callout and utilize pre-existing images without maintaining them in theirown communication apparatus.

The recording medium having a program for performing signal processingaccording to the present invention may be, as shown in FIG. 2 and FIG.3, a package medium having magnetic discs 61, 91 (including flexiblediscs), optical discs 62, 92 (including CD-ROM (Compact Disc-Read OnlyMemory) and DVD (Digital Versatile Disc), magneto-optical discs 63, 93(including MD ((Mini-Disc)(trademark)), semiconductor memories 64, 94,etc. recorded with the program for being able to be distributed to theusers as an individual component separated from a computer. To beprovided to the users by having already incorporated in a computer, theAbove recording medium may also be a ROM 42, 72 in which the program isrecorded, or a hard disc included in the storage units 48, 78.

Further, in the present specification, the steps for describing theprogram recorded in the recording medium comprise the processing to beperformed in time sequence in the order of description and theprocessing to be performed simultaneously or separately, not necessarilyin time sequence.

Furthermore, in the present specification, a system means the wholeapparatus comprising a plurality of apparatuses.

INDUSTRIAL APPLICABILITY

According to the present invention, it is possible to improve the imageprocessing speed and to reduce cost for service by distributing theimage processing to perform only the requested processing.

1-17. (canceled)
 18. An image processing apparatus comprising: an inputunit configured to input request information; and a signal processorconfigured to estimate a mixture ratio indicating the ratio at whichforeground object components and background object components are mixedin a mixed area according to the request information inputted by saidinputting unit, wherein the foreground object components representing aforeground object and the background object components representing abackground object of predetermined image data obtained from an imagepickup element having a predetermined number of pixels, each pixelhaving a time integration effect.
 19. The image processing apparatusaccording to claim 18, wherein said signal processor extracts desiredcomponents from the mixed area based on the estimated mixture ratio. 20.The image processing apparatus according to claim 19, wherein saidsignal processor separates the mixed area into the foreground objectcomponents and the background object components based on the estimatedmixture ratio.